0:00:06 | um |
---|
0:00:07 | today |
---|
0:00:08 | uh i'm going to share with you some uh results for that human assisted |
---|
0:00:12 | speaker recognition |
---|
0:00:13 | nist sre ten |
---|
0:00:15 | uh but |
---|
0:00:15 | first |
---|
0:00:16 | uh i want to point out that |
---|
0:00:17 | this was |
---|
0:00:18 | a possible only through the efforts of hard work |
---|
0:00:22 | uh of a lot of different people |
---|
0:00:24 | and so uh credit should go to |
---|
0:00:27 | uh jack godfrey and george got into |
---|
0:00:30 | as well as joe campbell uh our friends of the L D C |
---|
0:00:34 | uh and then |
---|
0:00:35 | um i can't think of it more as well |
---|
0:00:39 | so |
---|
0:00:40 | there's a question |
---|
0:00:42 | how how can human experts effectively utilise automatic speaker recognition technology |
---|
0:00:47 | um so this was |
---|
0:00:49 | um |
---|
0:00:50 | the perspective |
---|
0:00:51 | uh that |
---|
0:00:53 | uh we are coming from trying to answer this question |
---|
0:00:55 | uh when considering |
---|
0:00:56 | uh has a |
---|
0:00:57 | for sre ten |
---|
0:00:59 | um it's important to know this was a pilot |
---|
0:01:02 | um and also important to note |
---|
0:01:05 | that it was not |
---|
0:01:06 | uh what we hope |
---|
0:01:07 | it is uh uh informative and of interest |
---|
0:01:10 | add to the forensic community it was not |
---|
0:01:13 | uh design |
---|
0:01:14 | uh to be a forensic tests |
---|
0:01:16 | uh we |
---|
0:01:17 | did not use forensic data we |
---|
0:01:19 | uh leverage data that we're already using |
---|
0:01:21 | for the |
---|
0:01:22 | sorry |
---|
0:01:23 | ten evaluation |
---|
0:01:25 | so the hazard task |
---|
0:01:26 | uh was given to different speech segments determine whether |
---|
0:01:30 | uh they're both spoken by the same speaker |
---|
0:01:33 | uh has or was a subset |
---|
0:01:35 | of the sre core test |
---|
0:01:37 | uh there were fifteen trouser has a one hundred and fifty thousand hazard to |
---|
0:01:42 | a hazard wise human listening |
---|
0:01:44 | uh to assist in decision making and system descriptions |
---|
0:01:47 | um |
---|
0:01:49 | describe |
---|
0:01:50 | how much in this day |
---|
0:01:51 | uh exactly was involved |
---|
0:01:53 | uh participation was open to all who might be interested as it is |
---|
0:01:57 | uh recognition uh |
---|
0:01:58 | nist evaluations |
---|
0:01:59 | and we're just wouldn't strange |
---|
0:02:01 | uh from experts to |
---|
0:02:05 | so |
---|
0:02:06 | um |
---|
0:02:08 | when |
---|
0:02:09 | considering how much |
---|
0:02:11 | um |
---|
0:02:12 | data |
---|
0:02:13 | humans are able to process it |
---|
0:02:15 | fixed length of time |
---|
0:02:17 | um |
---|
0:02:18 | we determined that it would be necessary to try to select |
---|
0:02:22 | a few really difficult data |
---|
0:02:24 | uh in order to |
---|
0:02:26 | uh make the experiment |
---|
0:02:28 | uh challenging enough to be interesting |
---|
0:02:30 | and so the first question that came up was |
---|
0:02:32 | or even able to select |
---|
0:02:35 | data |
---|
0:02:36 | so uh we ran a preliminary experiment |
---|
0:02:39 | uh and this also |
---|
0:02:41 | a lot of success the protocol |
---|
0:02:43 | for the |
---|
0:02:43 | proposed as or study |
---|
0:02:45 | uh so what we did was we identified confusable speaker pairs from that's really |
---|
0:02:49 | for what |
---|
0:02:50 | um |
---|
0:02:51 | we use the automatic |
---|
0:02:52 | system |
---|
0:02:53 | uh i'm sorry a a set of automatic cyst |
---|
0:02:57 | uh from the follow up |
---|
0:02:58 | and |
---|
0:02:58 | for some voting system |
---|
0:03:00 | selected out forty seven pairs |
---|
0:03:02 | multiple cyst |
---|
0:03:03 | first |
---|
0:03:04 | and we listen to all the interviews for each speaker in each pair |
---|
0:03:09 | um |
---|
0:03:09 | they're these portions of the |
---|
0:03:11 | and selected ten pairs |
---|
0:03:12 | that's the most difficult to |
---|
0:03:14 | which |
---|
0:03:15 | uh from those we selected eight |
---|
0:03:17 | nontarget trials |
---|
0:03:18 | and then from the same speakers we also chose |
---|
0:03:21 | uh |
---|
0:03:22 | target rows |
---|
0:03:23 | uh something actually don't know now |
---|
0:03:25 | um |
---|
0:03:26 | that there's a little bit of |
---|
0:03:28 | but challenging |
---|
0:03:29 | terminology |
---|
0:03:30 | um |
---|
0:03:33 | so |
---|
0:03:33 | when we say uh |
---|
0:03:36 | target trials |
---|
0:03:37 | for model |
---|
0:03:38 | uh for |
---|
0:03:39 | test segment |
---|
0:03:40 | uh what we really mean in the case of target trials |
---|
0:03:43 | same speaker |
---|
0:03:44 | nontarget trials a different speaker |
---|
0:03:46 | and then for the model you might uh call this the known |
---|
0:03:50 | uh for the |
---|
0:03:51 | um |
---|
0:03:53 | uh |
---|
0:03:57 | suspect maybe |
---|
0:03:59 | uh and |
---|
0:04:00 | a case of the test |
---|
0:04:02 | um |
---|
0:04:03 | maybe the question |
---|
0:04:06 | uh let's see so |
---|
0:04:08 | um |
---|
0:04:09 | this every level here |
---|
0:04:10 | was used |
---|
0:04:11 | uh |
---|
0:04:12 | both training test |
---|
0:04:13 | uh |
---|
0:04:14 | four |
---|
0:04:16 | all of the |
---|
0:04:18 | uh |
---|
0:04:18 | has are probably expenditure |
---|
0:04:22 | the symbol fourteen human evaluators |
---|
0:04:24 | uh the uh volunteers were |
---|
0:04:26 | uh |
---|
0:04:27 | people involved in the project |
---|
0:04:28 | uh either |
---|
0:04:29 | um |
---|
0:04:29 | working for the government or |
---|
0:04:32 | they are |
---|
0:04:32 | uh as a data provider |
---|
0:04:34 | uh and people were permitted to unrestricted mostly |
---|
0:04:37 | table three |
---|
0:04:38 | test |
---|
0:04:39 | um |
---|
0:04:40 | evaluators provide a |
---|
0:04:41 | actual decision |
---|
0:04:43 | uh and optionally a coffee |
---|
0:04:44 | it's cool |
---|
0:04:45 | so |
---|
0:04:46 | i hear the results from experiment |
---|
0:04:50 | um |
---|
0:04:51 | see the overall miss rate was |
---|
0:04:53 | well |
---|
0:04:53 | percent |
---|
0:04:54 | the overall false alarm rate from four to six percent |
---|
0:04:57 | uh |
---|
0:04:58 | you see that on the left |
---|
0:05:00 | uh on the right |
---|
0:05:01 | you see the number of errors per trial |
---|
0:05:04 | uh you can see from the some trials are very challenging |
---|
0:05:07 | for example uh fifty |
---|
0:05:09 | five eight had nine nine |
---|
0:05:11 | eight years respectively |
---|
0:05:13 | two |
---|
0:05:14 | table |
---|
0:05:15 | uh |
---|
0:05:16 | three trials had |
---|
0:05:18 | where is that correct decision |
---|
0:05:20 | additional three have |
---|
0:05:21 | more than one third errors |
---|
0:05:22 | uh and one target one month or control |
---|
0:05:25 | there's |
---|
0:05:26 | no |
---|
0:05:28 | and this |
---|
0:05:29 | uh |
---|
0:05:30 | why shows |
---|
0:05:32 | uh |
---|
0:05:32 | number of errors by value |
---|
0:05:34 | a solid red or |
---|
0:05:36 | false alarm |
---|
0:05:37 | the outline |
---|
0:05:38 | yes |
---|
0:05:39 | uh |
---|
0:05:39 | you see all evaluators had |
---|
0:05:41 | false alarm errors |
---|
0:05:43 | i have had to |
---|
0:05:45 | yeah |
---|
0:05:47 | uh so |
---|
0:05:48 | the trials that probably be quite challenging |
---|
0:05:50 | and so this |
---|
0:05:51 | a preliminary experiment supported the idea |
---|
0:05:53 | that it would be possible to have a meaningful has evaluation |
---|
0:05:57 | uh with this use fifteen trial |
---|
0:05:59 | um |
---|
0:06:00 | when uh initially discussing |
---|
0:06:02 | potential for |
---|
0:06:04 | uh retina has experiment |
---|
0:06:05 | uh |
---|
0:06:06 | several |
---|
0:06:07 | potential participants were |
---|
0:06:09 | reluctant to do more than fifteen trials |
---|
0:06:11 | uh but do the concern for such systems |
---|
0:06:14 | if we can |
---|
0:06:14 | uh we decided to have a fifteen trial |
---|
0:06:17 | having fifty |
---|
0:06:18 | uh shock hazard |
---|
0:06:20 | um |
---|
0:06:20 | uh all the nodes |
---|
0:06:22 | they're still in |
---|
0:06:25 | he |
---|
0:06:25 | sure |
---|
0:06:28 | so let's talk about |
---|
0:06:29 | the |
---|
0:06:30 | has |
---|
0:06:31 | which in itself |
---|
0:06:33 | uh trans consisted of training and test speech segments |
---|
0:06:36 | as i said |
---|
0:06:37 | they have different |
---|
0:06:38 | the terminology |
---|
0:06:40 | uh |
---|
0:06:40 | trials |
---|
0:06:41 | where to be processed separately and independently |
---|
0:06:44 | um |
---|
0:06:46 | so uh what we did to help accommodate this |
---|
0:06:49 | was create a automated email system |
---|
0:06:52 | uh where |
---|
0:06:53 | um |
---|
0:06:53 | participants were |
---|
0:06:54 | so that the results for a given trial |
---|
0:06:56 | and then would receive a |
---|
0:06:58 | automated |
---|
0:06:59 | response |
---|
0:07:00 | um |
---|
0:07:02 | hopefully shortly after submitting their job |
---|
0:07:04 | uh |
---|
0:07:05 | with the lake |
---|
0:07:06 | to the neck |
---|
0:07:07 | yeah |
---|
0:07:08 | an extra |
---|
0:07:09 | uh unlimited amounts |
---|
0:07:10 | time permitting |
---|
0:07:11 | uh |
---|
0:07:13 | this thing was |
---|
0:07:14 | um was |
---|
0:07:15 | hello |
---|
0:07:17 | uh human listeners could in fact were |
---|
0:07:19 | in some cases other one person or a panel |
---|
0:07:22 | uh and uh decision and a likelihood score |
---|
0:07:25 | were acquired for each role |
---|
0:07:27 | uh_huh decisions could be made either from a combination of automatic |
---|
0:07:30 | uh processing and human expertise |
---|
0:07:33 | uh were made solely |
---|
0:07:34 | based on human history |
---|
0:07:36 | uh and we did not define a cost function |
---|
0:07:38 | four |
---|
0:07:39 | this evaluation |
---|
0:07:40 | uh and so |
---|
0:07:41 | this morning consisted solely of |
---|
0:07:43 | having this is |
---|
0:07:43 | false |
---|
0:07:45 | uh as i said earlier we us all difficult |
---|
0:07:49 | uh |
---|
0:07:50 | uh trials |
---|
0:07:51 | due to the number of journals |
---|
0:07:54 | and we did so from the mixture six corpus which was to be |
---|
0:07:58 | uh the training data was from interviews including various remarks |
---|
0:08:02 | and the test data was from phone calls |
---|
0:08:04 | uh |
---|
0:08:04 | put in some time |
---|
0:08:07 | uh to select the nontarget speaker pairs we're and he |
---|
0:08:12 | a speaker recognition system over four majors |
---|
0:08:14 | oh possible interview |
---|
0:08:16 | training of you test |
---|
0:08:18 | sex |
---|
0:08:19 | um i think an opportunity now |
---|
0:08:21 | two |
---|
0:08:21 | fig power |
---|
0:08:22 | uh how atlantic sea |
---|
0:08:24 | uh for helping us |
---|
0:08:25 | set up a |
---|
0:08:26 | uh speaker recognition system |
---|
0:08:28 | uh for the spring |
---|
0:08:30 | um |
---|
0:08:31 | after running the system over |
---|
0:08:33 | uh the four majors we identified |
---|
0:08:34 | speaker pairs |
---|
0:08:36 | uh that had a large number |
---|
0:08:37 | um |
---|
0:08:38 | uh |
---|
0:08:39 | very |
---|
0:08:40 | um |
---|
0:08:41 | hi |
---|
0:08:42 | scores |
---|
0:08:43 | uh and |
---|
0:08:44 | uh |
---|
0:08:45 | thus |
---|
0:08:45 | select the thirties and speaker pairs |
---|
0:08:47 | uh and |
---|
0:08:48 | from them as we listen to |
---|
0:08:50 | uh interview and |
---|
0:08:52 | uh |
---|
0:08:53 | phone call |
---|
0:08:53 | trials |
---|
0:08:54 | and selected |
---|
0:08:55 | nine |
---|
0:08:56 | uh based on perception |
---|
0:08:58 | uh that seem to be |
---|
0:08:59 | recent |
---|
0:09:01 | uh for the target rolls ran a full matrix |
---|
0:09:03 | of uh interview |
---|
0:09:05 | train telephone test |
---|
0:09:07 | uh and selected the actual |
---|
0:09:08 | trials that way |
---|
0:09:10 | uh |
---|
0:09:10 | thirty from a score |
---|
0:09:12 | and |
---|
0:09:13 | uh likewise by perception selected six |
---|
0:09:16 | just to be |
---|
0:09:17 | most assume |
---|
0:09:19 | after hazard to |
---|
0:09:20 | uh the first fifteen trials were |
---|
0:09:22 | uh |
---|
0:09:23 | the same |
---|
0:09:24 | as for has one for many another thirty five |
---|
0:09:27 | um |
---|
0:09:27 | we change some threshold |
---|
0:09:29 | to allow for additional speaker pairs |
---|
0:09:32 | uh we're trials but otherwise selected |
---|
0:09:34 | uh |
---|
0:09:35 | the data randomly |
---|
0:09:37 | are chosen |
---|
0:09:38 | oh we did listen for anomalous egg |
---|
0:09:41 | one |
---|
0:09:41 | this is |
---|
0:09:42 | speech or some |
---|
0:09:43 | uh |
---|
0:09:44 | uh |
---|
0:09:44 | and |
---|
0:09:45 | um |
---|
0:09:48 | uh so let's play a game |
---|
0:09:50 | uh |
---|
0:09:52 | yeah wisconsin |
---|
0:09:53 | same speaker different speaker |
---|
0:09:55 | um |
---|
0:09:56 | so |
---|
0:09:57 | uh |
---|
0:09:58 | here we're going to play |
---|
0:09:59 | that you segment |
---|
0:10:01 | and uh |
---|
0:10:02 | try to determine whether |
---|
0:10:03 | same speaker |
---|
0:10:04 | more |
---|
0:10:11 | oh |
---|
0:10:12 | oh |
---|
0:10:21 | oh |
---|
0:10:33 | yeah |
---|
0:10:34 | oh |
---|
0:10:35 | but |
---|
0:10:44 | should note |
---|
0:10:45 | no |
---|
0:10:50 | yeah |
---|
0:10:51 | oh |
---|
0:10:52 | oh |
---|
0:10:53 | i |
---|
0:10:54 | oh |
---|
0:10:55 | okay |
---|
0:10:57 | oh |
---|
0:10:58 | yeah |
---|
0:11:00 | yeah |
---|
0:11:02 | i |
---|
0:11:02 | i |
---|
0:11:04 | oh |
---|
0:11:04 | oh |
---|
0:11:05 | oh |
---|
0:11:05 | oh |
---|
0:11:06 | oh |
---|
0:11:07 | oh |
---|
0:11:07 | okay |
---|
0:11:08 | so |
---|
0:11:09 | how many people thought same speaker |
---|
0:11:12 | oh |
---|
0:11:13 | five different |
---|
0:11:14 | speaker |
---|
0:11:16 | okay |
---|
0:11:17 | yeah |
---|
0:11:18 | that we have |
---|
0:11:20 | sorry |
---|
0:11:23 | oh well |
---|
0:11:24 | yeah |
---|
0:11:25 | they rely zero |
---|
0:11:26 | okay and then for a child to |
---|
0:11:29 | oh yeah |
---|
0:11:31 | oh |
---|
0:11:31 | yeah |
---|
0:11:32 | yeah |
---|
0:11:35 | yeah |
---|
0:11:36 | oh |
---|
0:11:37 | oh |
---|
0:11:39 | oh |
---|
0:11:39 | oh |
---|
0:11:42 | yeah |
---|
0:11:44 | yeah |
---|
0:11:45 | oh |
---|
0:11:46 | one |
---|
0:11:47 | yeah |
---|
0:11:48 | oh |
---|
0:11:50 | oh |
---|
0:11:51 | yeah |
---|
0:11:52 | oh |
---|
0:11:54 | right |
---|
0:11:55 | oh |
---|
0:11:56 | oh |
---|
0:11:56 | where |
---|
0:11:58 | oh |
---|
0:11:59 | oh |
---|
0:12:00 | oh |
---|
0:12:18 | oh |
---|
0:12:21 | yeah |
---|
0:12:23 | okay |
---|
0:12:24 | well |
---|
0:12:26 | uh same speaker |
---|
0:12:29 | speaker |
---|
0:12:31 | yeah we have |
---|
0:12:31 | yeah |
---|
0:12:32 | uh so |
---|
0:12:34 | the uh first one was |
---|
0:12:35 | same |
---|
0:12:39 | yeah |
---|
0:12:42 | ah |
---|
0:12:42 | so uh |
---|
0:12:44 | twenty |
---|
0:12:45 | twenty systems from fifteen brave souls |
---|
0:12:48 | uh |
---|
0:12:50 | willing to |
---|
0:12:51 | uh |
---|
0:12:52 | dissipate a complete the fifteen trouser has a one |
---|
0:12:55 | uh and it's systems for six sites |
---|
0:12:58 | uh |
---|
0:12:58 | yeah |
---|
0:13:00 | i needed additional hundred thirty five trials |
---|
0:13:03 | answer to |
---|
0:13:04 | um oh sites that participated in has are also participated in the |
---|
0:13:09 | ministry ten about |
---|
0:13:11 | uh and the other side recruitment |
---|
0:13:13 | uh represented academic and government organisations from six different countries |
---|
0:13:18 | uh so for has or one |
---|
0:13:20 | uh here's the story |
---|
0:13:22 | um |
---|
0:13:22 | blue or |
---|
0:13:24 | uh target trials and read or nontarget trials |
---|
0:13:27 | uh and the darker colours uh represent their |
---|
0:13:30 | uh so at the bottom you can see sort of number of errors per trial |
---|
0:13:35 | and then to the columns |
---|
0:13:36 | you can see |
---|
0:13:37 | and this is |
---|
0:13:38 | false alarms |
---|
0:13:39 | for each |
---|
0:13:41 | each side |
---|
0:13:46 | okay |
---|
0:13:48 | uh missus corresponding information for hazard to |
---|
0:13:51 | uh just |
---|
0:13:52 | the information from the colours at the end |
---|
0:13:55 | um |
---|
0:13:55 | this is |
---|
0:13:56 | ooh |
---|
0:13:57 | also |
---|
0:14:04 | so |
---|
0:14:04 | um |
---|
0:14:05 | one question we are interested in was whether the human listening uh was |
---|
0:14:10 | uh worthwhile little selecting |
---|
0:14:12 | uh |
---|
0:14:12 | trials for the uh hazards power |
---|
0:14:15 | uh so uh are |
---|
0:14:17 | first look at this was |
---|
0:14:19 | um that the |
---|
0:14:21 | uh has one travels with the twenty systems had about |
---|
0:14:24 | thirty eight percent |
---|
0:14:25 | miss rate forty seven percent false alarm language wasn't too different |
---|
0:14:29 | uh from the eight systems or |
---|
0:14:31 | a hundred and thirty five trial so without L |
---|
0:14:33 | uh |
---|
0:14:35 | uh |
---|
0:14:36 | probably not worth the human life |
---|
0:14:38 | uh |
---|
0:14:39 | which was |
---|
0:14:39 | quite a bit uh to select the |
---|
0:14:42 | uh has or one draws weapons like |
---|
0:14:43 | same |
---|
0:14:44 | are sensitive |
---|
0:14:45 | uh we recently |
---|
0:14:46 | yeah maybe last year or so |
---|
0:14:48 | uh |
---|
0:14:48 | brought this out |
---|
0:14:49 | and do the same |
---|
0:14:51 | it's systems |
---|
0:14:52 | here we see the miss rate is about |
---|
0:14:54 | saying |
---|
0:14:55 | uh but the false alarm rate |
---|
0:14:57 | exactly signal |
---|
0:14:58 | hi |
---|
0:14:59 | uh so it seems as though maybe that has a one trials were a little more difficult but |
---|
0:15:03 | uh hopefully the hasn't you channels |
---|
0:15:05 | uh we're difficult enough |
---|
0:15:08 | uh this is |
---|
0:15:09 | yeah another view of the |
---|
0:15:11 | has a one has or two comparison |
---|
0:15:13 | uh the blue dots are |
---|
0:15:14 | uh has a while the |
---|
0:15:16 | red circles are |
---|
0:15:17 | or has or two |
---|
0:15:18 | uh and the lines connecting them |
---|
0:15:20 | uh sure there's the same system |
---|
0:15:22 | and all cases |
---|
0:15:23 | exception um |
---|
0:15:26 | i'm not sure |
---|
0:15:28 | um |
---|
0:15:29 | with the exception of |
---|
0:15:30 | this one |
---|
0:15:31 | system here |
---|
0:15:32 | uh |
---|
0:15:33 | and all cases has a two |
---|
0:15:35 | uh |
---|
0:15:35 | uh was less challenging |
---|
0:15:37 | i was or what |
---|
0:15:43 | and now this is a real obviously so uh |
---|
0:15:47 | they probably wouldn't |
---|
0:15:51 | and |
---|
0:15:53 | sure |
---|
0:15:53 | so uh |
---|
0:15:56 | with |
---|
0:15:57 | um |
---|
0:15:58 | has one |
---|
0:15:59 | uh and has or two we had some interest and |
---|
0:16:02 | uh how how to estimate also how systems did |
---|
0:16:05 | uh all the systems we were actually a little concerned that we're bias in this |
---|
0:16:08 | again systems |
---|
0:16:09 | since we use the system to |
---|
0:16:11 | uh select difficult trials |
---|
0:16:14 | um |
---|
0:16:15 | what we see here uh each red dot represents |
---|
0:16:17 | uh |
---|
0:16:18 | aside |
---|
0:16:19 | uh performance |
---|
0:16:20 | or |
---|
0:16:21 | uh has a one |
---|
0:16:23 | something |
---|
0:16:24 | to know someone brought up |
---|
0:16:25 | um |
---|
0:16:26 | and the last presentation that |
---|
0:16:28 | uh |
---|
0:16:29 | the dots on the axes |
---|
0:16:30 | uh actually shouldn't be on the axes |
---|
0:16:32 | and there is zero percent |
---|
0:16:34 | uh but |
---|
0:16:35 | that would otherwise |
---|
0:16:36 | be |
---|
0:16:37 | i'm not be visible so we |
---|
0:16:39 | put them there just a |
---|
0:16:40 | i see |
---|
0:16:40 | uh and uh so is is it the red dots represent the |
---|
0:16:44 | uh has or sites |
---|
0:16:46 | uh |
---|
0:16:46 | the lines |
---|
0:16:48 | uh represent |
---|
0:16:49 | top systems |
---|
0:16:51 | from |
---|
0:16:51 | uh the sre ten evaluation |
---|
0:16:54 | and |
---|
0:16:54 | thing |
---|
0:16:55 | uh |
---|
0:16:55 | is notable here |
---|
0:16:56 | uh is that in all cases the dots or or outside |
---|
0:17:00 | uh |
---|
0:17:01 | uh |
---|
0:17:02 | donna yellow line |
---|
0:17:07 | um |
---|
0:17:07 | so |
---|
0:17:09 | uh we do something similar what has or two |
---|
0:17:12 | uh and |
---|
0:17:13 | uh here we have all the same plot |
---|
0:17:16 | automatic systems |
---|
0:17:17 | uh and uh has or two systems |
---|
0:17:20 | uh the thick lines |
---|
0:17:22 | or |
---|
0:17:23 | uh automatic system |
---|
0:17:24 | and |
---|
0:17:25 | the lines |
---|
0:17:26 | or has a two system |
---|
0:17:27 | uh and we note uh |
---|
0:17:30 | a fair amount of separation |
---|
0:17:31 | uh between the two |
---|
0:17:34 | uh |
---|
0:17:35 | though these are the top |
---|
0:17:36 | sre |
---|
0:17:37 | uh |
---|
0:17:38 | automatic system |
---|
0:17:40 | uh and the slide |
---|
0:17:42 | we see |
---|
0:17:43 | um |
---|
0:17:45 | systems that |
---|
0:17:47 | uh or sites that participated both in house or two |
---|
0:17:50 | uh and in the |
---|
0:17:51 | uh sorry tell about |
---|
0:17:53 | uh and corresponding colours |
---|
0:17:56 | i represent the same site once again like lines |
---|
0:17:59 | uh are |
---|
0:18:00 | automatic systems |
---|
0:18:01 | the lines |
---|
0:18:02 | uh or has or system |
---|
0:18:04 | and the thing to note here is |
---|
0:18:05 | uh within colour |
---|
0:18:07 | uh the thick lines |
---|
0:18:08 | uh oh is closer to the origin |
---|
0:18:11 | then |
---|
0:18:12 | um |
---|
0:18:16 | uh so to summarise |
---|
0:18:17 | uh effort to select challenging data uh for has or was successful we feel |
---|
0:18:23 | uh we |
---|
0:18:24 | very pleased to that we were able to do that |
---|
0:18:27 | uh |
---|
0:18:28 | has or two trials uh only somewhat less challenging than has or what |
---|
0:18:32 | uh we can probably revises and say they were more challenging |
---|
0:18:36 | enhancer one though once again perhaps answer |
---|
0:18:39 | uh to was challenging enough |
---|
0:18:41 | uh for purpose |
---|
0:18:42 | this time |
---|
0:18:43 | uh performance of the hazard systems did not compare favourably with that of automatic systems uh on the other channels |
---|
0:18:51 | uh although i i feel obliged to remind everyone |
---|
0:18:54 | uh is dominant |
---|
0:18:55 | that |
---|
0:18:55 | there's |
---|
0:18:56 | uh there are so contrast |
---|
0:18:58 | that's |
---|
0:18:59 | um |
---|
0:19:00 | it's difficult to find any signal |
---|
0:19:01 | again |
---|
0:19:02 | uh |
---|
0:19:03 | in the |
---|
0:19:04 | uh statistical significance |
---|
0:19:06 | results |
---|
0:19:07 | um |
---|
0:19:08 | and it |
---|
0:19:09 | then there's a question uh if |
---|
0:19:10 | has our evaluations invaluable |
---|
0:19:13 | uh how should this be extended |
---|
0:19:14 | uh |
---|
0:19:15 | should the tense particle be changed or wasn't reasonable uh |
---|
0:19:19 | uh is there a good way to uh uh after the trial selection |
---|
0:19:24 | uh or wasn't uh also like was reasonable and |
---|
0:19:27 | uh what we keep on |
---|
0:19:29 | uh bumping up against |
---|
0:19:31 | this will signify |
---|
0:19:32 | so is there |
---|
0:19:33 | away |
---|
0:19:34 | uh to be able to get |
---|
0:19:36 | uh |
---|
0:19:36 | additional statistical significance |
---|
0:19:39 | uh without |
---|
0:19:40 | um |
---|
0:19:40 | burning the site |
---|
0:19:43 | too much |
---|
0:19:44 | and |
---|
0:19:45 | that is all |
---|
0:19:53 | we have time for |
---|
0:19:54 | some questions |
---|
0:19:55 | come on |
---|
0:19:59 | thank you craig |
---|
0:20:00 | um |
---|
0:20:01 | can you |
---|
0:20:02 | briefly |
---|
0:20:03 | clean |
---|
0:20:04 | oh |
---|
0:20:04 | what |
---|
0:20:05 | protocols |
---|
0:20:06 | oh |
---|
0:20:07 | i would be |
---|
0:20:07 | transmit |
---|
0:20:09 | uh what people actually do |
---|
0:20:11 | right |
---|
0:20:12 | hmmm |
---|
0:20:13 | sure so what i can say is that it it varied from side to side |
---|
0:20:16 | um and insights and uh system description |
---|
0:20:20 | um |
---|
0:20:20 | that was available to all the uh |
---|
0:20:22 | has approaches |
---|
0:20:24 | um |
---|
0:20:25 | in many cases |
---|
0:20:26 | that they were not even |
---|
0:20:28 | as with |
---|
0:20:29 | earlier in some cases |
---|
0:20:30 | and they um |
---|
0:20:32 | do you uh are or maybe more for the process of |
---|
0:20:36 | uh |
---|
0:20:37 | some uh |
---|
0:20:38 | uh process be online |
---|
0:20:46 | the question |
---|
0:20:54 | oh |
---|
0:20:55 | oh |
---|
0:20:55 | right |
---|
0:20:59 | i'm sorry |
---|
0:21:03 | did you produce |
---|
0:21:05 | they were just giving you binary |
---|
0:21:06 | so |
---|
0:21:07 | sure so um |
---|
0:21:09 | each uh has a participant was required in addition to giving the decision to give a |
---|
0:21:14 | uh |
---|
0:21:15 | score |
---|
0:21:17 | no that was i'm sorry yes but that was in the preliminary experiment |
---|
0:21:21 | yeah it was also on that and uh has or |
---|
0:21:23 | uh evaluation |
---|
0:21:24 | is |
---|
0:21:33 | we |
---|
0:21:34 | exp risk |
---|
0:21:35 | we we |
---|
0:21:36 | we |
---|
0:21:38 | making |
---|
0:21:38 | virtually impossible |
---|
0:21:40 | where do not exist |
---|
0:21:43 | bias |
---|
0:21:44 | it is |
---|
0:21:45 | or |
---|
0:21:45 | on a priori |
---|
0:21:48 | two |
---|
0:21:50 | humans are more |
---|
0:21:53 | three |
---|
0:21:54 | oh |
---|
0:21:55 | but |
---|
0:21:56 | where |
---|
0:21:58 | with |
---|
0:21:59 | fig |
---|
0:22:00 | Q |
---|
0:22:01 | well this is our concern |
---|
0:22:03 | um |
---|
0:22:04 | not a node in the case of the |
---|
0:22:06 | uh |
---|
0:22:07 | has or one trials |
---|
0:22:08 | i kiwi perception was also involved |
---|
0:22:11 | uh to select |
---|
0:22:12 | right |
---|
0:22:13 | so uh maybe it wasn't |
---|
0:22:15 | yeah |
---|
0:22:16 | i |
---|
0:22:17 | my suggestion |
---|
0:22:19 | two |
---|
0:22:20 | right |
---|
0:22:21 | right |
---|
0:22:22 | right |
---|
0:22:22 | wow |
---|
0:22:24 | her |
---|
0:22:25 | to use |
---|
0:22:26 | you know |
---|
0:22:27 | speaker I D |
---|
0:22:28 | yeah |
---|
0:22:30 | two |
---|
0:22:31 | oh |
---|
0:22:31 | right |
---|
0:22:32 | thanks |
---|
0:22:34 | yes |
---|
0:22:35 | fig |
---|
0:22:35 | a system |
---|
0:22:37 | two |
---|
0:22:38 | picks |
---|
0:22:38 | right |
---|
0:22:39 | sure |
---|
0:22:40 | i |
---|
0:22:41 | something |
---|
0:22:42 | was |
---|
0:22:43 | we have to |
---|
0:22:46 | i like to be |
---|
0:22:53 | you you said that one of the reasons of doing this is to see if the combination of human |
---|
0:22:58 | a machine |
---|
0:22:58 | would help |
---|
0:22:59 | yeah |
---|
0:22:59 | she just |
---|
0:23:00 | for a considerable |
---|
0:23:01 | oh faster |
---|
0:23:02 | did you see |
---|
0:23:03 | uh persists for people have a system |
---|
0:23:07 | the normal needs to you know |
---|
0:23:09 | and then if they want to answer |
---|
0:23:11 | yeah |
---|
0:23:11 | a hundred fifty trials |
---|
0:23:13 | they could listen |
---|
0:23:14 | so you can decide |
---|
0:23:15 | which ones |
---|
0:23:16 | oh |
---|
0:23:16 | fig |
---|
0:23:17 | be helpful to listen to |
---|
0:23:19 | i |
---|
0:23:20 | and then |
---|
0:23:21 | um |
---|
0:23:22 | you can listen to whatever we |
---|
0:23:24 | just a decision is just a score |
---|
0:23:27 | uh |
---|
0:23:28 | of course then you wouldn't have seen from sri system but you have |
---|
0:23:31 | you combine |
---|
0:23:33 | decision |
---|
0:23:34 | where |
---|
0:23:35 | something which you have |
---|
0:23:37 | buying so |
---|
0:23:38 | yes i think that would be interesting to think oh no |
---|
0:23:40 | uh however is that some sites did |
---|
0:23:42 | their goals |
---|
0:23:43 | you know human machine fusion |
---|
0:23:45 | uh |
---|
0:23:46 | so |
---|
0:23:46 | i guess we |
---|
0:23:47 | um |
---|
0:23:48 | are able to get |
---|
0:23:49 | uh |
---|
0:23:50 | some very small glimpse of that |
---|
0:23:52 | uh but i like yours |
---|
0:23:53 | efficient |
---|
0:23:55 | also |
---|
0:23:56 | you |
---|
0:23:57 | cool |
---|
0:23:58 | so attribute description |
---|
0:24:00 | during the walkthrough we look from you |
---|
0:24:02 | you too |
---|
0:24:03 | you you man |
---|
0:24:04 | be |
---|
0:24:04 | system |
---|
0:24:05 | you know the true system |
---|
0:24:07 | and we are trying not to answer two questions |
---|
0:24:10 | try to evaluate you just a few men people human expert |
---|
0:24:14 | and try to |
---|
0:24:16 | this time |
---|
0:24:17 | you selection process |
---|
0:24:19 | you |
---|
0:24:20 | maybe not |
---|
0:24:22 | you do |
---|
0:24:24 | if we try to to see if we are to |
---|
0:24:27 | system |
---|
0:24:28 | it's good to evaluate remember |
---|
0:24:30 | no |
---|
0:24:31 | you |
---|
0:24:31 | we must maximum |
---|
0:24:34 | yeah |
---|
0:24:39 | um so i just one and |
---|
0:24:42 | that um of course are |
---|
0:24:45 | it's |
---|
0:24:45 | i |
---|
0:24:45 | yeah |
---|
0:24:46 | table |
---|
0:24:46 | what |
---|
0:24:47 | we |
---|
0:24:48 | we're more interest |
---|
0:24:49 | see |
---|
0:24:50 | she |
---|
0:24:51 | no |
---|
0:24:51 | you |
---|
0:24:54 | um |
---|
0:24:55 | so |
---|
0:24:56 | in mind that |
---|
0:24:59 | we're going to be establishing um |
---|
0:25:02 | in additional mechanism for |
---|
0:25:04 | communicating about |
---|
0:25:06 | uh |
---|
0:25:07 | future evaluations along this line |
---|
0:25:10 | how |
---|
0:25:11 | um |
---|
0:25:12 | for a couple |
---|
0:25:13 | or |
---|
0:25:14 | in |
---|
0:25:14 | right |
---|
0:25:15 | um |
---|
0:25:16 | what might |
---|
0:25:17 | no |
---|
0:25:18 | in future english |
---|
0:25:20 | to help mode discussions and whatnot |
---|
0:25:23 | just as |
---|
0:25:24 | there was just one |
---|
0:25:26 | group for |
---|
0:25:27 | story continuation |
---|
0:25:29 | uh will you |
---|
0:25:31 | well thing |
---|
0:25:32 | where |
---|
0:25:32 | he's or |
---|
0:25:33 | oh |
---|
0:25:33 | yeah |
---|
0:25:34 | see dover downs |
---|
0:25:35 | and |
---|
0:25:36 | suggestions for |
---|
0:25:37 | how to make this |
---|
0:25:38 | oh |
---|
0:25:39 | an interesting um |
---|
0:25:41 | evaluation in the future uh |
---|
0:25:44 | and |
---|
0:25:44 | many very interesting |
---|
0:25:46 | suggestions including multi |
---|
0:25:48 | we |
---|
0:25:50 | and |
---|
0:25:51 | related activities like you can |
---|
0:25:54 | yeah |
---|
0:25:56 | where do we |
---|
0:25:57 | like |
---|
0:25:58 | well |
---|
0:25:59 | it's |
---|
0:25:59 | and uh |
---|
0:26:00 | um and you |
---|
0:26:01 | contact me your |
---|
0:26:02 | nist people |
---|
0:26:06 | because |
---|
0:26:08 | that's cute |
---|
0:26:08 | a fusion between human beings |
---|
0:26:11 | and the automatic |
---|
0:26:12 | to to see |
---|
0:26:12 | but |
---|
0:26:13 | well the human part |
---|
0:26:14 | it's something |
---|
0:26:15 | commissioner |
---|
0:26:16 | didn't |
---|
0:26:18 | yeah |
---|
0:26:18 | i didn't have one |
---|
0:26:19 | sure yes uh we have not |
---|
0:26:21 | uh |
---|
0:26:23 | we're not really looked at this |
---|
0:26:24 | uh |
---|
0:26:25 | no i find it very interesting |
---|
0:26:27 | part of |
---|
0:26:28 | uh |
---|
0:26:29 | part of it |
---|
0:26:29 | challenge is |
---|
0:26:31 | um |
---|
0:26:32 | that in some cases |
---|
0:26:34 | uh the human and |
---|
0:26:36 | um |
---|
0:26:37 | uh machine |
---|
0:26:38 | uh results were |
---|
0:26:39 | use prior to coming to us |
---|
0:26:41 | so would be difficult those cases that is out |
---|
0:26:43 | what |
---|
0:26:44 | part of that |
---|
0:26:46 | percent |
---|
0:26:48 | thank you |
---|
0:26:49 | Q from |
---|