Speech Transcript - Jack Godfrey and the Art of Evaluation and Benchmarking

good morning this is joe campbell from mit lincoln laboratory

and i'm joining by tell me canyon in from university of eastern finland

were here this morning with a set of distinguished panelists

to honour and represent and reflect on our dear friend i jack godfrey who passed

away last one

we have a very full agenda today so

i think what will i wanna just remind to remind you of a couple things

there were manning many people wanted to enjoying the panel but on the virtual stage

door only room for six

so if you could send messages in the chat

we will be happy to share them with the godfrey family as a memento from

this event

now let me handed over to tell me

totally that's a moment of silence

okay so probable system also might be so we will go moment was on as

norm so

based on

okay the criminals

so i think you know angel

thank you if i can have the next slide please

doctor john j godfrey led a very distinguished career

main so many friend friends along the way

and is a pillar of our community

he is largely the reason why this workshop exist

through is

efforts in leading corpus space research

is you know phd from georgetown in linguistics

his early career work focused on psycholinguistics at wright patterson air force base that the

air force research laboratory

and the university of texas at dallas is collier centre that some of the earliest

papers in human speaker recognition where published by jack in those days

he was a contributor to the revolutionary speak and spell troika texas instruments

this was

the first that i know of spelling computer for children very effective

is served as the first executive director of the linguistic data consortium

and

this enabled him to drive a whole corpus based already community

back any all day is

people will publish results and whatever datasets they would collect

usually ad hoc

without rigour in it was impossible for people to compare performance across systems

so this was a big contribution

as the chief of the h l t group it and the say you provided

national direction in litres and brought together this international community promoting an evaluation culture as

whereabout to see in a few minutes

is a dedicated has been father friend mentor

and loves to fish and hold great stories we learn a lesson

i just have another minute or so the

but what i'll do you now is let tony at a few remarks as the

is gonna s p l c signature

thank you so his or her or something the special interest group of a speaker

role recognition community and so we are going to some this is the remainder of

this on is a common is able to so are you over to work well

known

really wanting to introduce bills

with control

thank you tell me

i'll just quickly say the that's jack with his wife nina the aim at wright

patterson air force base

he loved to fish

that was is real big thing and then where we've got a huge smile there

is a holding my bell i'm two hundred centimetres tall so he was upon is

tells for that great picture

now i'd like to introduce a movie where jack will be able to tell us

in its own words

the importance of evaluation and the scientific method and its quest for understanding

this is courtesy of the johns hopkins university c lsp animal lecture he gave in

two thousand twelve

but i have the video please

thus

she

it is

you

sorry

there

right

see

really

it is

once i models

right

it is

this is

three

for so

well

here

okay

so problem

landline

yes

although

right

one

so it is

right

why

right

okay

one

use

say

this

also

these

that was

then

right

yes

thus

yes

more than

right

okay

why the

the only ones

yes

seven video really captures the essence of jack godfrey is methods in driving an international

research community

and

finding results there are surprises and wanting to understand why

and this was a key the johns six jack godfrey six s

and largely to our community six s

those evaluation curves over the years

showing performance the problem getting harder making performance gains again changing the problem to make

an even more realistic was this of virtuous cycle

the jack brought to the government and in the us

and also internationally and help the number of government partners and research t means

there was instrumental in the advancement of or feel

the things that have been made by the community here at odyssey over the last

two decades have been remarkable and i congratulate you offer that and we can think

jack for supporting of the datasets to drive all this and is this insistence and

sign

now will transition into the upper panel session

we have sticks distinguished colleagues in dear friends joining us

and the first one is craig greenberg

he was actually in that video with jack that you just a the whole video

is over an hour but we didn't have time to play the whole thing

so let me handed over to craig take the of a few minutes four minutes

worth the of remarks or show a slight evaluations

great

and it was a degree privilege and you're to get to know and we're project

jack was instrumental in having various human language technology evaluations of news

including of automatic speech recognition spoken dialog systems

and of course of speaker and language recognition evaluations the median and ninety six and

can do you want to their

definitely involved in the nist evaluations offering his vision an insight sometimes aligner french

and all along with a joke in a kind word

and resulted in two thousand and i started working the nist on the speaker and

language recognition evaluations which we typically referred to as sre and lre

actually all the direction of all aspects of the sre use another use

in the beginning evaluation training data collection is all the weighted including workshop

there is a leadership dedication

generate householder real of the sre series and the first evaluation with ten or so

systems and around four thousand trials

into a series that commonly includes hundreds of systems could be millions of miles

the other easy reason to and similar groups that are is all sources

actually

i understand the timit corpus the jet worked on was originally set around one male

and one t

before he retired we began running evaluations entirely online

surely this tremendous contributions and it was remarkable progress of the evaluations and the technology

and so

part of whitney objects contribution was brilliance as a linguist

just to give a sense is

sometimes i feel f voicemail he'd come in here office and right on your whiteboard

the phonetic transcription of your message

where we used that's what i was told by some of those of her colleagues

personally i really think of and all that if you didn't answer

i didn't that didn't get to see the sort of thing lyman action

they're in one been you notice my use of the non-lexical conversation also

the sound was a

we just found that using able to correctly infer that i was the simpsons fan

furthermore

i'll i'm not sure of me is that only we also really like the simpsons

i'm so grateful that in addition to working with great i got a known person

we don't think what we were years after evaluation maybe millions and at conferences

where of course we talk about language and language technology

but also politics history philosophy economics education

just to name a few topics about which is usually knowledgeable

another so much

you reminded me not only professional managers but also personal items that will continue to

carry with me all my

and so like to take this opportunity to take two injectors mentor shipping guidance as

well as for is tremendous contributions the human language technologies

and to the nist evaluations

after such monumental where

i don't we all agree that he rest in peace

i'm

thank you craig that was very nice

next we have douglas reynolds

who

is an assignment in the us government and the is in my

don't can i join us in person but he kindly sent this video the we

will play now

recording could happen or weekly where p e

three would like to select keywords and i'm your friend happy jack godfrey

but my earliest recollections of jack

with the precursor to this whole odyssey workshop series ninety nine core margin use what's

one

is relatively not you researcher having some of the jack stature come to my coaster

listen to my work asking questions make me feel that what i was doing with

the most important thing but the major motivator might really career

but jack kind of way of doing that with whomever speaking with

you matrix will since they were the most important thing but is warm apple nature

one of jack's

legacies to the speech community is being a high here in strong component of common

corpus evaluation driven research and development

in today's your numbers benchmark datasets and challenges

it's harder to imagine a time when even small data sets were expensive and harder

to manipulate

and there were numerous speakers were published and cried datasets using a tricks

and tracking enough everybody had about a hundred percent accuracy

this around the mid nineteen edus the darpa was one of the first organisations to

really push common corpus evaluations so the technical progress could be meaningfully measure

jack along with this texas instrument colleagues which is george doddington you can strong advocates

of the value of common corpora evaluations

jack was instrumental in recreation in collection of the first conversational telephone corpus

call switchboard

but in a standard is pretty small twenty five hundred conversations with five hundred speakers

but the time it was one of the data is that really started should drive

the fundamentals in speech recognition and speaker recognition

jack and became the first executive director and linguistic data consortium

where you continued his dry to have a common corpora for human language technology tasks

in expanded their catalogue with numerous amounts of

novel weighting the collecting speech corpora including text into that mix and other types of

data sets

after ldc

jack went on to become the each individual t research the dod

here you continue to abdicate can drive you called virtuous circle research development that allowed

him to leverage the worldwide research community

but each antony open speaker and language recognition evaluations run by nist

these evaluations have been

the driver from much of the core technology that we'll be hearing about obviously this

week

jack was always a fierce contractor the people in his great wherever you worked and

not much one to deal with bureaucracy very well

in fact idiot is good at here is always the skunk at the garden already

saying we need to be standard and not trying to put on the niceties pursuit

circuitry

in addition

jack's professional accomplishments he was also warm

affable charming mare

and i wonder person around

in the problem family man friend mentor in a heck of a storyteller

is it your soul this greatly by discriminately and based family

thank you joe

therefore

thank you do that was really nice integrity is one of the words the

there really stands out

so yes i want to introduce john for us all born this

to speak about jack's work mostly in their intersection in the world of four and

six

and then all hand over the introduction of the following speakers to tell me

john first law

and stroll

good use and my slide or died we

if you can do it that would be great

in less they have them at the odyssey team

there's actually go have your slides

you go is it possible

okay sure

you

well

we next

well i i'm very are known to be here and to say some walls of

the

but to dye my story reading

be began to be in a nineteen and ninety eight even if i saw him

in matching you for years a the before

nineteen eighty eight that was my first time in the usa

my first time in the nist

it's very

the first five actually

trying to do speaker verification and more exactly the first time i

the idea of the evaluation with this war

and from my own to go to public

a common the vocal

and

to be able to

and the wheel beginning my first the discussion we draw

where was

very successful for you to my

english ability to two hundred and speakers it possible to and it can also but

also i will the dark object that and

a little bit rate and very great by both

was already a big i in the community and also by be context

it was not at all

easy

the for this for during this first the meeting i

have a strong memories about discussion within a but what means that you identification and

that i and

i have in mind if i but i agree on the table problem of via

a cost to

no one runs we can try to say what is happening you in france at

this moment

and a duck

this then to us

not only but refundable

and the and the conclusion of the meeting

and nineteen ninety eight meeting came back and i and say okay maybe we have

more on this removal

currently convert remote quite three you and whatnot

the mainstream or what we walk

next slide please

i'm

very happy because after each time i had the problem but we need to use

the to meet and

there are more rules interaction with him

i'm not sure because my initiation improved a lot but stephanie i

i was closer to him and more able to

to discuss the without fear of with e

i remember particularly discussion about

the speaker or reason for forensic an investigative it application of the freedom to be

that is we started with the

do so drawing me many

and also very

no i discussion in two fifteen sixteen when he

came to me after the presentation of my paper with the way the g

the ball really video speaker characteristics and he say go

i would like to continue all are not going to start the phd unto the

able with the

what we things we have now be able to start with the shah really mixing

linguistics aspect and the speaker recognition so

this is a very nice

moment in my brain no

i will be of individual to after on the maybe

we can extract the and of my

mostly consist

because when they look at the at the video of jack had to the you

know restore i realised

how much necessary and i

had a huge influence on my career and the on the what

you why you research

so i we just

say i but

for to what i really don't from him what they share with limited to about

some more best and i don't know stuff like a wheelchair would be we will

stand me and we show me for a long time and remind conferences and my

which were

a and i mean you could give to me you okay to say that will

be but

okay like to some friends well so well we use the or to general solution

to propose something about perhaps the previous or some of what is an okay mister

thank you can you show lights lies please the

i was the organiser of that workshop in nothing is that the dog and of

course of ready for the two and it was my first meeting these with jack

and the

wanted to

to say

what i feel about the influence for his work on the things we've done in

europe so that score to the next slide please

where you will see the next slide we will see the party seep than

and

many people that you know that contributed to these workshop be nodding e

and

what i want to stress is really what is on the next slide

so if you go to the next slide

basically

and the times there was a the caucuses the which is a quality needing only

t on the speech database

and speech assessment

and

that was before this work shall mean id for whatever swearing that jack that

at the same time that me conclude that any he was already

the british in this a knuckle full database

which was the first the only for speech database that was produced

and

i mean in this

this database is a five thousand speech is

that the

say the control the sentences on the

an and the sun the all sorts of items of fifty items i guess all

the telephone so it's it was the first the green the database that inspired the

some of the word i was doing in a in switzerland that the time where

we record the swiss french polyphone

database which was equivalent to the microphone

and another database that we call clearly well she's

for a speaker variability so that we had the about the fifty speeches that were

able to already called several times the of the polyphone at a

so that we had the intra speaker variability as well

in the meantime jack actually are produced also lizzy voice of cross is tiny can

eric which was the at nazareth the database

and dior ideal that is the key lingual effort

was really is the start of the european project called speechdat

which was

the started after that

actually in nineteen ninety five variability

and

which are created the

databases for

most of the european

other languages

and also

i guess and

database for

speech over a that in america so there was a

these so that project

and of course that was

all the work that jack has been has started

with the switchboard was taken over in the

ldc for and down that for recording

but as a date that this is like fisher and mixer and many and that

was also the in pure initial the inspiration for the not data collection which was

the non dairy in than entering polyphone version

so that was really the start of the or the work he has done from

the for the microphone and the voiceover hispanic and eric out was really spread all

over the world

for these collection all the polyphone databases so i think and like to

to recognize or the work that has done in the in pushing for all of

those these the data collection and very good and the price

so that the next slide is not i know there is an ex like but

it's not very

it's and

it's just the picture from the microphone

data collection process but it's not very interesting

so that's that was my conclusion

okay

thank you very much

okay you're a monster

so do you might wanna be a really global or anymore is able to texas

instruments on performance sensible things

list model

you you're for only one session together and part of it

i'm gonna

a couple of other memories for a second one what everyone right contribution

and evaluation dataset to remind right

i wanna from the other side or

formerly on and more recently as well and you know

i have privilege working with amount of money are really far i see your out

there somewhere

and you only other directions are i like

is he's

and what i remember work in different areas of course and

i remember being test the artifact them for right i guess in a straight into

your all incidents down

now the neighbours and

so anyhow the songs and i think many of us have clearly so

finally in which are commonly used and all of us engineering or c

interest in our culture you in line with one

and that of my first experience that

i also had privilege of the in one of the corporate he recorded and that's

not

but what i remember that is the one for restaurant on the unit was and

i think if it is needed in the southwestern style right now

i don't know everything analysis mathematics and so question here

and then we enjoy the moment in many years that about a fantasy effectively now

at sort of early on where i had no and then a more recently proceed

with very or a and

so i'm gonna start using and not getting having means and is the low say

might expect that he was a little bit from the writing it is not someone

who really

and i think it's time in industry and government

influence

no the since the disaster the problem is the really i what he'll say is

this is the only thing we have for one

and had a strong interaction with that

and you would say some through the l is the okay for another center agents

easier

i think the prefive a little bit differently fancy expectations about the really wanna make

it anything we system

towards a shot the problem

we had a zero eer from last the listening for us and hang out with

the solver

we are very grateful for what you percent has a and b and sort of

driving

evaluations and matrix and is a fast

for the

and then added to retire the on our seamless the list and z and with

higher bic your after the somewhere

and sort of went back is true and channel four and it's each time and

you are zero one myself

i to have them with that but we're so much humour these days we did

what we call this the science thing with a

so that sort of with the poles are comfort obviously start it is true academic

in the data and as a science than that in

you the and you don't these models and for us and

yes i second what everybody saying was it is a realistic in two

where in different

i think you're emotional

so maybe scores in an somehow some and also something remotely sensed on the influence

with two

exactly

and they're in general form

for organising the session

to say that seen me so we started session here

i don't like i was sitting across from jack daniels

a unique words in building a small words of in my understanding the other understanding

of problems in jacket this way you are kind of talking with people

that was very

but it's following but also in a collaborative way never and i really am stay

on the you know is kind of understood that this was kind of a room

and a collaborative effort to train two million four

so i'm just a few words in step two gently for students in for the

academic field i think time i pointed out he's done in the nn exploration move

into the government industry in i think he's perspective on the academic field

where is kind of somehow influenced after going to industry into and into the government

no set yours a so

you know no it just started to exactly mimicry of georgetown is usually inverse text

a to do a postdoctoral position of the you are still for star airspace medical

research lab and that actually i think about a lot more on physiology aspects so

objects work inside into there's linguistics background

and after that he joins in a year is in texas at dallas here and

extension here's your you g d in slope maybe it'll sciences a most of is

that what was really focused in the carter centre color centre was found nineteen forties

actually by e

are three founders a ti johnson examining bringing this river system for one for johnson

which actually the person is found colours ten and they are just one for is

non speech production him psycholinguistics

i talked many quotes is there a he did we eventually real eh jean of

researchers

including a number of errors and i know just actually used a question medical centre

and the dollar sign an age project that show that just actual ear as essential

has extent to actually what it also is effective utility to discriminate in class prior

phonetic features in speech this is

well it is the m zero is gonna work understanding the science and how the

signs actually motivated maybe somebody technology advances in should feel so it's balance or something

that which really i think that justin but also something good okay and it gave

people there are more the technology side better understanding science

the forensic science to better understand in technology

well as you but jealousy worked there quite a bit with a lot still min

ross perot's ram or it would be

a good move a variety of the that researchers in also supervised belongs to a

joy still which measures usually student actually to follow him to g i would be

shifted or gi

a i thought we were still min it's this week a one of is what

collections is that usually there is to a most systems everybody will be more clinically

oriented towards a phonetic speech science

well to begin well we use asking questions and variance fiery by jack's was the

man is one of your

i when the engine is related you gaussian random scroll celebrating is twenty fifth anniversary

we had set up distinguished lecture

process and be projected the you distinguished lecture for twenty fifth anniversary okay

and subject you know being however

having a good sense if you my estimated i need a picture you chair

and in suggest that well you know the only actually that could you know really

in that room and there was a picture of him holding no more the largest

fish i've ever seen any insisted you might and you use this picture you know

for the for the keynote and i

we had with him as i felt you know it's probably something that i was

gonna get five from what might be in my question so i meant that a

mormon and you know you really like to use this picture again that are using

a better than you see here is reached i and

just gave expensive two days with us okay

in october two thousand eleven is connotes on which i think backwards i think we

can buy light of the five minutes you i auditorium the engineerings for we based

on and you met many students in one of things that their students are is

appreciated was

both this knowledge is worse than this way of balancing

i guess so it

say that

just understood forms of workout not

say

something were words and say always

five value or whatever it's a

then it was it is gonna of one of the digits are you had news

meeting with very well one of our are all stock staff members just probably and

we've been with us for many years and i got it should holding this one

sets for

cover mechanisms of speech by german by name alexander graham bell and just and where

you initiated or its science and technology so maybe my common to be jot notes

committed to helping students in the academic community

you know listing and yes reason lr use as well as i cast in interspeech

is gonna talking be collecting gerard injury have also pointing out that you know you

make it make a way to go to those jurors to a one on one

language and you know it's fell

you had his undivided attention your attention

we seven interactions there now see you really understood have to a inspires to actually

motivates to understand that they were great opportunities for them in your rear and

a new helping thing one many students are two very productive right

in one of us to be used as you got can

it plays and gender probably and i had the weather spending for they were generated

four hours that numbers i just two minutes for our own and we found this

one

i think

well kitchen that was kind of like the whole mirrors in jack a we do

we show in a look at things from different perspectives and you can see a

lots of comments there but i think doing that the a

pretty much everywhere we went with me

given some historical knowledge of a few you know but i can and if knowledge

of just

everything historical

the motivation for the online specs were just not robust one something jack in every

aspect

so maybe just in closing house data

well if you look at the nist and so is now we competitions it certainly

challenge speech community

really motivating from many researchers and teams and feel

a competitions also offer extensive opportunities for common speech used to just very speech speaker

and language or your wrist can so i think one of the looking at evaluation

and benchmarking

gender effort seems to have also used

challenges and stories now read

really one three inspirational wine for maybe students feel

i'll say he of course need strategic fusion of speech science and technology it is

also inspired motivated in mention countless young researchers to

in the linking enhanced or where you know someone there really just wanted to make

sure where

if you want away from conversation jack one layer a better person your value understanding

in more

suggest a with a little snow you for so many years

and you would be remiss fire mainly are probably weeks

well most importantly all the friends and family if we can contact with

i hear someone okay

what we wanna

it remains for everything you contribute to the fuel and since the site

apparently remotes some it's been really inspiring to are used or is one of the

analysis

something to remotes for one on my husband's to margin

this

so this work we will go over the main point

i don't see people have baseline well you know

and war conditioning

only

so as you for your decision for my only a single file and you probably

all whereas if you john godfrey contribution size and sufficiency argument

you can even

his personality was already okay a few

an amazing what you vector and one more community and we all

i've got you can partly by one or more one issue as an are you

gonna

for each iteration fisher

where q our are actually in a more efficiently

especially if you expressed if you have some experience here fishing

you know as a few us what are you last year

are contribution a legacy our fear

well i

wow

it will be naming policy option bass or more

after

younger of three this year

you know go free so that or exactly strong i mean or feel and how

she by one norm asking you about

and what jack on a and

what action

and it would be and he will be greatly missed

well this all one would allow someone okay be one or i mean

i don't we all remember

john asks more well assuming you see the first winter we see john best or

a one-best or

and all that if initial something infinite all five k i don't know that what

really

it won't one question okay you

thank you very much machine appreciate the language too

finally and we would like to invite brian godfrey

to give some words and the half of the godfrey family and what are not

there it is to have brian representing the family today right

based on or something everybody

from a sitting

this to you put this together and fusion the on the in our family accessing

alone

fusion

as my doesn't mind noted and an outstanding memorial on jack was a renaissance man

i and different on about on that were results just randomly

but i guess i can i losing everyone standard but multiple languages

you know have a creation and technology good as we talked about here you talked

about his stand academia industry

all snotty little excursion appliances buildings and outdoors

comprehension literature history economics along the successor

listed is balanced analysis on

unrelated reasons you could be easily areas but sometimes you can be

smart analysis and within the to unit this study

not of everything adjustments is that it

and then sawdust only

outstanding

well brothers not master

the background model

as was ordinance no rates on integrity

a generalist really fair and open minded readers

all things it will for reverb discussed

already

thank goodness of my brother sometimes also instantly stations

right along all about that stuff also

he's into an unlimited amount of energy

and the decision was done on doing twirl one something

you data manager originally you don't and others

which embrace from this time

regions and store managers and a right

really and still an intense desire to one and then do things with the knowledge

three task one of the common

it seems i noticed and the nodes are received one or more of the people

posted

was only focus on the mentor

it only once the best comedy the received one i one time student and years

later

note that you don't and then the best teacher

well after reading

no memorials and you're not everybody set here

i realise if we see that in a long history stuff

teaching be a professor so i'll be use the influence you have also values calling

yes i was bruno was applied in the real us to really implementing was

in that also struck me

the speakers and politics not all the

so many names of people that i never had the opportunity need in detail about

how incredibly smart people how it was addressed as far

but what they were able to do

and speaking with you several million over the past months replenisher

so no the feelings mutual

most a stalling so

also you know i held everybody them also saw perfect embodiment model professor

no surprise i can be

first renew noisily that's where you know where he is waiters or was that was

is wrong

so now on the side of the challenge

and right you know use an and just you know

it's only many times you were used for the next question else

and since all about what else is a year

unless personally i don't know anybody more

and well i don't know the specifics

i do not exhaustion

i know the speakers

usually always closest friends intolerance it only

what he brought of the field of human language technology will make a little better

place was no less

enclosing and not accumulating better

almost close to the minimum orders of that's where in the streets enslavement

just as a simple sentence somewhere for describing then

and the loss is the last time

you know personally and professionally

so again thank you guys so much

for including

myself and my family in this and

really means a lot so

you

thank you very much brighter

that's very touching the here from the family and

your father was an amazing

i'd also like to thank the panelists and the attendees

and the odyssey teen ugh on a

key go and talk a for me for helping us put this together and my

colleague tell me can in for

all the organising their you did together with me

i wanna remind everybody to

at end of the upcoming virtual tore of tokyo that starts at nine fifteen uses

a different link

so be sure to

this came from this and go to the page to see that

thank you that looks really nice and it'll be

there appear after this session i also want to point out the

tell me what's gonna point out the

the other session go ahead tell me

no nor is true

also going to one and also show your work has been really on a normal

to remove organs in this isn't so is reasonably

suppose you are

yes so are very well as well as on the eval or someone benchmark a

two minutes or something so on sore everyone so the turks influence on the on

the on benchmarking

so based on this one to one

so this summer

okay maybe this is what channel

yes thank you can only i think

the other thing i wanna dimension is

there is a session and evaluation in benchmarking chaired by omit from nist

the duh directly follows

in the frame of jack's leadership

and guiding the community that's a and the man and them and session so you

can view that at your leisure

do you please joining the tokyo tore and that's all we have affected a i'm

sorry were four minutes overtime

thank you everybody i really appreciated

well

thank you

and smell on anyone

Jack Godfrey and the Art of Evaluation and Benchmarking

Live sessions