Speech Transcript - Cogl: Having fun with GPU graphics

say hi everybody thanks a lot for coming on to my talk about global

so i'm robert right guy work for intel open source technology centre on the kind

maintainer of the whole project

so for those are not familiar with probably it's other consoles gpu graphics library originally

created for the cluster projects they are these days as a standalone project

it was created to deal with the possibility across different versions of opengl so course

opengl yes opengl and even experimentally a bit these days with the web you know

it's is so dealing with portability across operating systems

one of the differences with some other graphics if you refuse are base is that

initially when we created it we were interested in supporting more multiple sort of orthogonal

components being of the contribute to the graphics rendering in a way that didn't trampling

each other's us take tracking

so in a still come hoping to go through first a run through of how

we submit and process geometry with a gpu and when the something out

and i want to then give

an overview of the utilities in the couple library that try to make it really

practical for writing graphics applications

be on just access to the G P your hardware and then talk about some

of the specific things we've been developing more recently just say that people get some

insight to how the project is actually developing

so first so wanna talk about

basically the lowest level api even in cover which most closely matches have a gpu

process is data

i'm kind of personally interested in trying to expose more people to understand have a

gpu itself works especially in the context of user interface development where i think we

try to be on the utilise the flexibility of this hardware

and the state-of-the-art for you are is today this is quite basic in comparison to

you can see to sort of game developers and how successful they are using this

hardware with many of this and that kind of constraints is us with

interactivity and real time graphics

anyway the first thing that any applications gonna have to do they're gonna use google

to draw something are with the G here is great and of the cobble context

this is gonna just you could cool context news is gonna create your sandbox for

all the data the call was gonna need to track for your application

so then we're gonna need to create a so it's a destination for what we

draw remove are rendering this is called a frame buffer

a frame buffer is a collection of to debug is

the but we care the most about is the colour buffer and that's where we're

gonna store the results visually of what we're drawing but a frame of mind also

contain other ancillary about this so that are is

for accumulating the depth of the geometry that is used during this that we have

to see so you know these guys

in probably create a frame buffer you can you can great and on screen frame

buffer that would be visual to a visible to use of coding got one screen

you with the size alternative you would create an offscreen frame buffer by creating a

text you have with a given size and then you would rat that texture with

as an offscreen frame buffer

so if we have a destination that's where we're gonna store results then we need

some data that we're gonna process with the G P A and how does a

gpu except data

and gpus

also actually the is so the thirty forms of data so we i think

the in cover we support lighting image data have for

sorry

in the order

and image it's a as a form of a so that we would expect we

have person a gpu be on search german tree and it's a in probably support

lighting textures on the from finals from afar name or from about four with a

given size that and by sending data on that texture us one is being up

to images into a larger at this of images a

and it's often useful to be able to

how can we just again as a high level protection the way you often it

is a compact and to be have the packs more images into a larger images

to avoid state changes and another form of textures the gpus process of pictures great

from when your system resources such as X eleven expects or while and

i'm sorry the older this

okay

it's just that

the left and right click it doesn't very this is a real andreas breaks the

slide down to the sections which of vertical sections like this pressing that right skip

saver

pretty big chunk that so which

so about a so this'll spree this slightly better slide talk a week or two

meeting date to the G P so depressing we need we have to do is

the actual geometry into the G P A so first we need to actually understand

what they to the G P processes and that is geometry which is test selected

from triangles is the primary form in which we price estate on the G P

A so you

to understand

you have the how we describe astrology the next thing to understand that's a this

up about that is the stuff like just jump ahead at and sorry about that

but we're gonna fast go through explaining how the geometries this but apologies and

so the way we describe it geometries might tell is german treaters trying plated from

attempt to assign different triangles and there's triangles themselves made of that but to cease

and that is that sees are themselves components of what are called attributes

so examples of attributes he would be so positions for each vertex in a triangle

so you would have a position

which could be a coordinate two D or three D coordinate reach but text in

that geometry or it might be a colour for each vertex in the geometry or

it might be i

a texture coordinate that would could be at to detect record there

it is the mapping a specific subregion of an image that is a don't you

there's a lot of flexibility in the gpus to what data you can associate with

about season your geometry and these are just very the examples but it's you can

have all of the cost and data so i see the outcome actually

but they don't you can also have many different apologies so tip about apologies would

be that you have a triangle list which is where you would have

so i sets of three but the sees which represent completely disjoint and disconnected triangles

or you might have a strip of triangles and that would be you divide your

first set of three but sees feel first triangle and then one additional verdict is

gonna connect to the last two that sees of your script to make one more

triangle and so one more vertex connects the last event these answers a more data

efficient way of describing some geometry than with the less if that's applicable to your

which on your trying to describe and last example would be a band where you

just have a single but clicks in the centre and then the rest of the

triangles gonna spend a lot of that

and that's and one

so how do we actually described estate using the called I P I

so the first thing we're gonna do is describe

decide what kind of data we want to associate with the each of those matrices

there is some different strategies as to how you present estate to the G P

so i'm just gonna give you one very typical like how we would do this

we decide that we wanna have a position the colour and texture gordon associated with

each vertex in our geometry we're gonna create a since even greater about six am

i-vector extractor in this case and it has a position it has two components of

the position X and Y coordinates into D and there of type fly we're gonna

have full components of the colour but we implement out for a and each of

those is a by a number gonna have two components protect recording it and those

are touch like nist one

and

this but what about an this is just that we've taken on my vertex struck

which is a vertex i'm with declaring an array of these and we're gonna plot

out we've got three about sees for one triangle and with plotting out the positions

of the but sees giving each method the colour and where specifying the texture coordinates

you

so we nine to up like that data to the gpus able to read and

prices that and that's just a case of uploading into a cover that should be

but we do that by coding a battery but when you repulsed in the size

of the data that we want to apply time we pass in the actual point

to the data

and that's gonna give us a reference to about

the next thing is that we're going to have to describe the G P or

how is it that it can what we got data needs to know that i

have to that data so it's right three the but sees to process it

and so if we reach out to be we need to create a cover that

should be and we go global attribute new and we're gonna pass in a reference

to the buffer which is a date there are actually so we do we declared

and we're gonna give each attribute and nine we're gonna specify what is this be

and the number of bytes that we need to step to be have to move

from the best the text of the second but sex and so on

and then we're gonna need to specify what's the offset from the beginning of the

buffer whether gpus gonna start reading for that should be here to

process that data

lastly we need to specify the number of compared to data have you have to

take

like two components

so assuming we do the same thing for the colour and texture coordinates we have

for batch we recovered attributes now

we can now define one or more primitives

the referring to that should be right so

so at this point this is where we specify the actual topology that we want

to that we wanna gpu to use to interpret that data so lists triangle friends

trips or whatever

so with the funny primitive here way because it's just a single triangle that apologies

pretty meaningless so we specify that there are three but sees the belong to this

specific primitive

and then we specifying the actual attributes the belong to this primitive and in this

case we using this of our function takes a know to make the list of

that should be says

so the

other thing that gpus comprises besides the john geometric data that i just described that

is that they can also still image data as textures citizen where we were a

bit outflow of their own and i started going through

how in cover we can like text data so we have we can let them

from files we can load a texture just as an empty but and fill out

from the C P with the point to we can also create these textures which

are cool data structures like an example of a high level texture the pose able

there are numbered if i don't use

but it's sometimes useful for this band of packing images one texture

to avoid

like a changes for

you are use cases

and might these pictures from when the system resources

what is

and

so we probably talk we put off right but what we're gonna store results with

what our data which is these triangles and images as textures so now we need

to figure out how to describe the pipeline by which this is all gonna be

processed

so i'm gonna give a really sky high level view of any it example of

how you can configure gpu pipeline to process data of modern T is pretty don

flexible and they're a number of stages a cut optional state is really but that

i'm not representativeness example but it is still a very typical a common example is

for is this technology

they're gonna start with our geometry we coming in on the left

and we're gonna bust going to this but expressing stage in this stage we're gonna

transform a german three we're gonna scale it rotated translated we're gonna projected so that

points at a button away from the origin are gonna attenuate towards vanishing point to

the

realistic

and at the end of the stage all about german tree is gonna be transformed

into the coordinate space about frame but

and it's conceptually a range within a scene of other geometry

bands

the next stage after that is the frogman processing stage in the G P is

gonna go through each of the triangles in our geometry and it's gonna i would

like that with the pixel grid about frame buffer in it's gonna consider for each

pixel where the centre of the

what's called a fragment where that is within the bounds of that triangle then it's

gonna run a program to basically calculate colour for that fragment at this point when

you're calculating not colour

you can but that's of those textures the we floated

and

also the coolness of each triangle where we have those attribute values coming in there

is can be interpolated say that for every intermediate fragment network processing we have interpolated

attribute values to contradict

program

so in the final step

we take all of those fragments and we know need to humanise there is onto

the existing contents of the frame buffer

to build up a whole scene and it this point we might also be reading

back insisting colours in a frame buffer to be able to say if the effect

of transparency by combining in coming

destination ones

and so the buttons wasting the from present processing state is there is programmable stage

is very flexible the planning stages tends to what is a much more limited functionality

done and fix function of us still a can lay down do that stuff

and

and the well in which the actually the hardware prices this is very paralysed there

is if you have a primitive without one for triangles coming and there's not really

any bordering guarantees as to have a gpus gonna price essays triangles

and it's gonna be doing lots of geometry the same time and similar with a

fragment processing the order exact order in which is gonna priceless fact is not something

you get to guarantee and it's very paralysed that you will be doing

sorry

so kind of have a gpu prices to state a high level and

i don't have the time to explain lots of examples of using the cover if

you have to describe a

variations on configuring E D U but just to introduce you to the couple plot

line object which represents that whole processing part by the G P A

and we can create a peaceful pipeline by conical pipeline you

and to just give one example thing of for figuring it where we are introducing

a texture into that pricing but we're gonna associated texture with what's known as an

idea and this idea of layers use during the fragment pricing stage where they can

be combined and

us associating attention with layers there of that

so basically now we have all of the bits that it takes to actually draw

something a process similar gpu i'll just mention frame buffer of data a primitive we

got a processing pipeline we can call primitive draw busting the reference to that data

that frame buffer and is that the choice of five line

and so that's basically that's basically it but introducing datum pricing G P A and

it's

it is to distill as much as i got time to be able to

i give you

so the next thing i wanna talk about the utilities at all the whites that

go beyond just that lower level access to the gpu hardware that trying to make

it actually really practical to get stuff done is just basic things a little mice

all graphical applications need and gonna start off by introducing the math utilities that come

with bubble

we have matrices api matrices of the way that we deal with describing transformations of

geometry in combining those transformations

just essential stuff really matrix not build on top of that capability and kate to

the number of applications that want to organise that the geometry into seeing profs

so it's very common that you want to basically start these transformations together is interested

in seeing trough pushing them onto the stack "'em" popping them as you move around

the scene rough and this can say you re accumulating lots of transformations about this

is the on changing i frequency

and that is it just

you know use of all kinds of things the next can be to have a

you know it's on and dealing with that is

rotations there are several rotation types that we have been a couple user an example

of a compact representation rotation comprised of three orthogonal rotations

and quaternions or another format it's an encoding of an axis workstation and they have

the nice property being able to interpolate really smoothly between the have the pros and

cons and but especially with quaternions

they're pretty some pretty fancy matt's behind the it's not

every time you need

and colour utilities so being able to convert from rgb take you saturation like two

spaces

other

which i just snippets this is the way that we experienced the programmable parts of

the program will parts of the gpu

this

this design say that you can have multiple components contribute into the i roll processing

of geometry and it so cooperating within one framework so

the way this works is that we have this idea that points in apart line

so examples would be vertex processing stage where you wanna be able to

make some it's you want to build on the people processing of probable it but

introduce some additional transformations for example you can provide a small snippet of code which

is using the G L of cell shading language to insert into one of these

have points in a can either replace that whole

section replace any people processing

or complete and or pen so that it can cooperate without science

and similarly for fractal processing like and set to

and that supplied to

and looking up textures and there are

places as well

clipping lots of applications once we have to restrict

the range where they want to restrict where the influence of drawing within the frame

buffer

to a certain to D regions and

so if in cover you can click to ask align rectangles transform rectangles this interactive

a battery primitives to D pass describe with fancy is awesome lines

and similar to the matrix that we consider the you have seen rockies cases where

you want to be able to start these on top of each other is interesting

trough and effectively region is the intersection of all of that is the company push

regions

rectangle patching especially when it comes to user interface is a currently very dependent on

using tech should rectangles as the

so building block of most interfaces and because gpus are not really very effective a

practising very tiny primitives one of time just to send a couple of triangles for

what within separate draw close is

is not and make good utilisation of the of the G your hardware

so it

with this there is the if you well within certain limitations i for instance with

if you little your textures into atlas textures

so they can all be referenced as part of one two like one month one

primitive

and if you limit yourself to certain types of clipping then in a lot of

cases actually couple can batch almost like whole scenes of rectangles into a single goal

cools

it's quite useful

so don't have a bunch of optional sup libraries that come with problem as well

i mean for instance we have come up anger which enables us to rend text

laid out with that anger make you know i

we have hope a which provide a test like to have a teapot so describing

shakes with bessie is lines and curves and being able to get a retains object

representing that estimation of that and being able to redraw that without incurring repeated incurring

the cost a test selecting that geometry

how would you nist two thousand is to

easily integrate a just

regular gas to carried into the a couple application and managing the state

between nice

to it you know it's

i would use T is a recent addition which i i'm speedier integration

so couple based on the G stream a framework

so now ago three some of the much more recent things that we've been developing

just say you know idea of where the focus is a and where it's going

so that's the yes i thanks to find a at the beginning of this year

we introduced a couple frame info right you know i

and it's this is about being up to get asynchronously extra information about frame is

completed so for instance when we went to remove

also possible frame of the impostor to be displayed

and it takes some time before that can post a response and gets that frame

actually displayed on to the monitor and we'd like to know too well what's the

time stamp when that really hit the display and get that but we have location

so that it can be a bit smarter about our rest is animations

based on the prediction or when that frame is really gonna hit the physical display

see you have these

i think

thanks to kind of stay in for contributing offence compensate pi two call this allows

us to insert marcus into the come on stream that was sending to the G

P A

so that we can have a bunch of what that we're interested in knowing when

the G is finish processing this way

since when that work is finished then we can get a call back into the

application to be able to

i respond to that and the use case they had this was for video integration

where they wanted to be have to make sure that

but the gpu had finish any something the need to do a certain textures before

was handed back to the video decoder library to avoid conflict

thanks to i mean a man a lot of one of the interns so we've

been working with recently and saxony a lot that's the supporting i'll at this work

for G S T

on speedier integration to couple and it's it works

particular well with to pay particular attention to making sure that it works nicely with

the state i the eyes and with multi texturing side it is a lot of

freedom to

i will all kinds of your own custom affects and things that will be involved

in a rendering of a video frame so if you wanted to do from processing

like be saturating contrast brightness that kind of thing and trade is you don't wanna

have intimated rents we listing so you need a cooperative system to allow you to

combine these effects

one

and as you have the use cases this was for project called break sure

mention that

so what you have is

another thing that were exploring at the moment there's a project that where this using

couple that i'm working on who rate which is i you are rendering engine and

provides interactive tooling for designers for improving the web design where were exploring more advanced

uses of the G P are in the context if you i development and it's

we're interested in supporting a number of homes and even we're interested in exploring supporting

web applications

it's an expiration price my remember where i will be possible and okay

so what's next

so something that i always stripe in the way that

in terms of maintaining the project and driving a project for this is ensuring that

the feature what we develop

it's really driven by the projects work the requirements of projects are actually using so

we don't wanna be planning architecture astronauts just coming up with fun stuff like that

we wanna actually have a reason why we develop any particular feature so the ounce

really this question is to consider the people using the cover like my

the for new uses a couple will take it and you directions

so some of the current users a couple that implements it

we have a lot to obviously are the do the first user couple

and it's even today the that's a influencing certain things for instance the high dpi

stuff so i guess they will need to make some small changes to the called

tango at are to consider a

the size of clips that we tool

but there is a special use case of a couple especially when we consider the

recent web two wraps

name show that way to support and i'm show

and that the bonds a bunch of the feature well as well in kabul

well who is not something i'm working on activity at the moment but it so

i wanted to mention it just because i did put quite a bit of work

into this a few years ago and it was an interesting project it really do

that should drive quite a lot of

thank you think over the time

and this was adding a gpu acceleration back into card was taking a different approach

to carry G L and are generally have said making some different tradeoffs

but it did some interesting things to do if retaining the translations of parts and

trying to be a little bit okay well about using the fix function possibly gpu

for gradients and

the be a bit more efficient

i would say much about you best "'cause" i guess not many people here about

that but you know robert's writer back and for E best the

this is like a couple years ago we played around this is yet not really

got serious use case but it was interesting the time because and there's actually some

slightly nifty things that you best dates to do with matching actually which we were

interested

"'cause"

and so a much more recent projects that working on this week which i mentioned

just briefly it's a okay and i don't hide for this project to be quite

a big driver a couple going forward actually

that mission here is to really explore in the context of user interface development how

we can use more features of the G U and how we can also optimized

the web fly between design is prototyping but at the moment without many constraints this

to the technology used in the end and

creating interactive tools

the hell optimize are what are even connecting we to sort of devices i network

and such

things

and

right about that

big driver

so those of the project striving that's night it sort of update for the projects

and so i'm entrance it's in it if anyone's interested in this kind of thing

graphics processing and it's especially a

why and this audience in the context of user interface development

i'd love to invite people to learn more about the G P A

and contribute to call i've either directly or through downstream projects

and creating a more that downstream projects and we

we called wants

so join us about join the mailing list get involved in discussions during this and

i have see we

created a channel for call dedicate the corpus the other day about time really just

the program

trying to send a message on again off together and say for us on better

if you just want to i project and there's a website now

find what you what you will we we're gonna probably try and i mean it

but the labour office i dare easy

which will time

nine what to make it as easy as possible

that i'm sorry

any questions

gonna task

the this building

what this piece to me

well are you use the

even if you're using name show the patching that it provides a would make it

Cogl: Having fun with GPU graphics

Developer Experience Track

Robert Bragg