All right, hello, my name is Robin, and I'm joined today by my co-workers, Timo and Valer,
we're from Elman's Voip team.
We work on Elman's call mainly, and today we're here to talk to you about Matrix RTC,
Excuto, a Battle Royale.
Yeah, so of course Matrix is an open standard for decentralized communication, and it
is divided into rooms, and in these rooms, users can send these persistent events to send
text messages or things of that sort.
But of course, this is just one kind of communication.
There's a whole world of real-time communication that we'd also like to be able to do on
Matrix, such as exchanging voice-in-video data, or maybe real-time data for a virtual
world, or a whiteboard, or something like that.
So yeah, what if we don't want this persistent event model?
Well, we've spent the last few years trying to come up with a concept for Matrix RTC,
which lets you augment your rooms with new features, so it lets you add new slots for
real-time communication to your room.
And so the possibilities here are that an admin can come in and add a slot for, I don't
know, your Saturday night group calls, or your 3D virtual world, or your multiplayer
crossword game, whatever you want, and so these all are things that get added to your
room.
And yeah, that is the idea behind a slot here, is that it is a combination of an application,
which specifies what kind of data participants are going to exchange over this slot, and
an identifying tag, just so you can have multiple parallel calls or something.
Yeah, and so m.call is the very first application that we are proposing to add to this
spec, and this is, of course, just the basic functionality that you need for any voice or
video call, but third party apps are completely possible too, so you could just come in
with a custom identifier here for your world simulation, and yeah, how this works is an
admin will come along and they will send a state event, and this is how slots are recorded
in matrix.
There is stored in room state, so yeah, the admin here would send a state event with
the application specifying the type, and there can also be some applications specific
data there, and the reason why we are using state events for this is because it allows
for matrix R to C slots to be moderated essentially, they can be opened and closed just
by the admins, you can decide exactly what can happen in your room, yeah, and that way
they are persistent and they are authorized, so yeah, the idea is that just like you join rooms
when Allison Bob wants to get together on a call, they will connect to the slot within
the room, and so how they do this is Alice is going to send a sticky event, and sticky events
are another new concept that we are proposing to add to matrix, the deal with these is they
are just like normal room events, except they have guaranteed reliable delivery for basically
up to an hour, so whenever Bob comes along and sinks his client, he is going to receive
Alice's sticky event even if it was like the scroll off the timeline, and yeah, so the nice
properties of these sticky events is that they are a femoral, which is appropriate for
these RTC session participants in a way that state isn't room state, it's like you can't
delete an entry, so it sticks around forever, and they can be encrypted even, and they're also
low privilege, you don't have to be an admin to send one of these because they're not a state event,
yeah, and the most important thing here is that Alice's event declares how you can re-ter
the RTC transports, yeah, and so how this works, well of course in reality Alice and Bob
aren't a little Lego many figures sitting in a room together, they are people with computers
sending data across in that work, and so the way that Alice and Bob end up exchanging their data
is they are going to choose a transport, and the transport that we've now implemented for
Ellen Call uses life kit media servers, but the idea is there could be more added to the
specification for sure, and these are always going to be either resources provided by your home
server, or maybe some peer-to-peer thing, so yeah each member will choose their own transport
that's given by their home server, and then we let them, yeah, just upload their media there locally,
they decide they're on transport because we want to maintain the federated model that we
know and matrix even for real-time communication, and so yeah each member here will then go and
also download the media from the others from MoT server, and so this seems maybe a little unusual
for a calling system because you have to connect to multiple things just to participate,
but the reason why I think we love this model and things, I think it deserves to be in this
fact, is that it handles large calls gracefully without forcing them to be centralized, so you get
a nice balance with scalability, because the thing is with large calls, more often than not,
many of the users will actually end up converging on the same home servers, that's just how large
calls in like office environments end up working, there will be resources, but like a home server
provided by the company that most users are on, and so they'll end up sharing transports,
and that means that to connect to this larger crawl, well okay it's only of seven people,
Bob only has to connect to just two servers to download the media here, and yeah so the life
kit servers are still handling the fan out of all of everybody's media, which is the hard
part normally for large calls, if you have to upload your media to and other people, that gets
pretty pretty expensive for phones that are on on flaky mobile connections or someone on a DSL connection
yeah, and so those are all the basics of matrix RTC already, so right there are four concepts you
should remember, we have slots, this is how you set them up and moderate matrix RTC,
the admin sets up the slot in the room by sending a state event, and that state event specifies
an application, which is the type of data you're exchanging, and then to join, we connect by sending
our membership event and we're publishing our media over our chosen transport, which is described there,
so yeah, slots, applications, membership, and transports, now there are a number of other things
that I haven't been able to describe here today because I don't have that much time, the entry
interesting stuff is the late events and notifications and encryption, but there are a number of other
talks from last use matrix conference, if you're interested in learning about how matrix RTC works in
detail, you can go check those out, and yeah, the MSC's for all of this are well, yeah, they're all up
on the matrix spec proposals repository, and I think they're not quite ready for this spec
core team yet, perhaps, but yeah, they're very close to being sent off, we're making our final
adjustments to them, and yeah, so they all need your feedback, please go take a look at them and see what
you think, all right, and we'll have to team up. Yeah, thanks for the introduction or like the
main part of the talk, and thanks for giving us the background on the matrix RTC bits, and now we
at the point where we want to build something with it, so with this like big changes or not big changes
but this evolution of the spec, we also did some really big changes to our co-based to make it all
compatible and have a proven implementation for the spec core team, and why we did this, yeah,
refactoring basically, and adaptation to the new spec, we noticed that we will eventually have a
very clear cut in our code base, which basically gives us something like a matrix RTC SDK for free,
and that's what we will talk about now, so how to use it and what can you build with it.
So first, this is like a very, very rudimentary overview of what our element code code base looks like,
we're building on the matrix JS SDK, and those are all the parts, which are just purely matrix
related, sticky events, and state events, and then we have the element code repository, and this
thanks to this refactoring and adapting everything to the new spec, it's now also pretty clearly split
into two sections, the bottom section is matrix RTC and the top section is that I'm really
element call, so the bottom section is things like connecting to life-kid, connecting to multiple
SFUs, because that's all not done on JS SDK, and we noticed we can easily pull out the
bottom block and expose it as a new built target, so how does that look like, basically here the
sline is just in between the layer between the yellow and the green block, and that is the API
we decided for this matrix RTC SDK, so it's like pretty small, fairly simple, you have things like
joining, leaving, tracking members as a observable, so you see anybody, any new joiner and
lever, and then of course you have the super important functions like sending real-time data,
sending matrix data, and inside the members you have the connections to them also,
yeah, subscribe to the data, and we even have like a high level data observable for all the
life data, and now what we are actually now going with the demo is that we use this SDK target
and build like a very, very simple HTML template application for the good game and then,
so you can use all the matrix RTC bits, like connecting to life, good connecting to multiple SFUs,
handling delayed event, handling sticky events, authenticating with matrix, all those bits,
extracted away and you just call a weight, create matrix RTC SDK, you get an SDK, and then with
the things we just showed, you can say store the data observable into something called data
observable, and then you can use that in godo, so godo has like a pretty nice
FFI basically for JavaScript, so you use the JavaScript bridge, get the window interface,
and then do dot matrix RTC SDK, what we just stored in the window, and then with this SDK,
we have access to our data observable and all the other things, but let's just look at the data
observable one, and we can use this in a game engine to build games with it, and that's where
pass over to Valera showing you. Hello, so my name is Valera, so I'm going to show you like the
good at part, so the part that probably will fail. So yeah, because we wanted to show like everything
we do in matrix, basically we build upon blocks, building blocks, but you can reuse to do more
things, so this sticky events you could like send event that is additional delivery going to
super important to say, I'm in the call, we have the delay events, so now you can do schedule event,
I can send this event, we send tomorrow at 10 am, you can do self-destruct event, but we want it to
do a game, so this is a full local setup here, I'm going to do like that first we're going to try
live after, but you can see already that it's a very federation, so they're not running on the same
home server, and so the first part is, this is like this part is matrix communication, so you can
see the other side, say join game, so I have now a game, so here but everything that happened was
in the room, so I've got a aware of the membership, now I'm going to do start, but the start is
going to send events by at a data lifetime layer and I'm going to send the jumps, so I can only
jump for the left player, so one of them will fall down, so you should hopefully see me on the
other side, like you can see that I'm actually, so this is like using two different live
keeps transport, and now the matter of this talk, we created a public room, I mean maybe
Timo and I would start a game, so the room is here, it's flat P-OTC that falls down, so
I have to try from desktop, I don't think you can from mobile, there is no which it supports yet,
but if you can, we will let this room open a bit, if you want to try a bit later,
let's, let me first start the game, some waiting for players, so that's the part where we see if we have good
network, it's like you stuck, so okay I have a demo, I mean I've recorded something,
so it's like, when we started that I say how hard can it be to have a world-but
world-back network could, because I wanted it to be like really real-time, I wanted the birds to
fire bullets, things that make you do small jumps, so I said what, world-but-net will hurt, it's easy,
but here it was with a bad network and you will see that the other side is like
world-back and catching up, so it's teleporting, but yeah that's how it looks like with two people in
so let me go back and see, oh it's three players, okay that's the time
for it, let's try it, oh what, which one I mean, okay okay, so it varies too much like it's
going to last the players and dead, so yeah I've had basically the game, so you can add that, so
I think I would push about to get to other people want to try it in some way, if you went to
other break, I don't know who he's top one, other five players, so very important player more,
so there's also another game, I just want to show you, so it's a wee jet, so basically I can go here
next-tension, we have another game, I won't be able to type, but because I'm holding the microphone,
but just to show you that now it will ask you for a wall of permission, it's a bit scary,
so like the widget want to take control of your life, so it should be loading,
or maybe I already loaded, I don't know, oh this is the wall, okay so there's a lot of legacy
havens here, so that's something we need to work out in the, we need to update, to just
give you a bit to group a bit to permission, so this, yeah, this is the last bug, it's requesting
that several times, okay, yeah you can, we do have a couple of players there, I can again,
we don't buy a team, oh you just have to press it as often as it called it, okay, so here
the game's like you have to type, or I have to do something, you have to type to, or you have
to start the game right? I think it started, yeah, so everybody just types if they are not sure
what exactly, they can look at the three-letter preview on the bottom left corner, and then we have
multiple people on the track, I think for a layer there, I don't know, I'm a little, oh my computer
they are sure, there's a little good audience, self of course one, I'm at a to fail,
yeah, but as soon as one person will, will make the full rounds, it will send into the room
how quick they run, it looks a little better on this one, yeah, should we just let go of this?
Oh, that's the last time we tried to play, I can try to reload it, it's hard, yeah, I think I already
worked one time, I mean, a bit, I'm being this one, maybe it doesn't like it and they're out to you
of them, I think I already clicked on that, oh I can see people now, oh there, okay, okay,
how much am I clicking here, oh I'm not clicking, oh yeah, maybe that's okay, so you have people
racing, you race by typing the letters, I'm trying to go, you could have put some real words, no,
yeah, so that's it, so like this is a bit on top of the matrix RTC, just to show like
all the things you can build with the SDK, so any questions?
And when I only have a JavaScript SDK, if you think of having all the SDKs, just just for
the fake one to make all the games, all the languages, for example, you know,
and use this real time for like, and then I have, and you can use it, I guess I have to use it
after doing my own SDK and top of the, I mean, you have to do that sort of, I repeat the question,
so basically if you want to do a game but using something else that you're doing, yeah, another language,
so you just have to have that sort of bridge, so as we say in God it's quite easy, like I,
they have a JavaScript bridge, I mean, just follows me to get a,
I don't know if you people can see things, but I'm just getting the bridge, I'm getting the windows,
and I'm getting the match starts, so if you have that bridge, then it's just Mario of listening to
you haven't, and you can do the game in JavaScript, and I think we'll just work with it.
So I'm not doing it in Java, that, and I might only say, one to use the Earth, yeah, we have
for more to do something, I'm not going to go for example. So I can use, I mean, in
general using, which is not going to be able to, I'm JavaScript, I mean, so I can use it,
you guess, and you have to be my own SDK to connect, oh yeah, probably, if you cannot use,
any of the GSSDK, you have to build someone coming on some benefits together.
And I think there's another dimension, or, yeah, I think there's another dimension to this
question, so basically the way we develop RTC applications currently is always as a widget,
that has a big advantage that you don't need to give some random game, your full matrix
account with all the encryption keys, but instead element web, or whatever other trusted
widget supporting client you have, we do all the encryption and key back up and everything,
and the widget can just talk.