SC2/D3 speculation thread

[iago]

Eh, I don't know that that's true.  Depending on the structure of the client handler, the server could potentially send more data than the client handler expects.  For example, supposing they send a packet with header length of 40, but only 25 bytes is used (by the version defined by older clients).  Newer clients can recognize that the actual length of the packet is longer and know to take some action with the rest of the data.  Nothing has to be done with special markers or anything like that.

If you do that, and you change the protocol a couple times, you end up with a bunch of stuff glued to the end as well as a bunch of stuff that's undefined. It turns into a mess. Look at SID_CHAT_EVENT -- half of it is blanked out. And what if different (future) clients require different things (like the way War3 requires a server signature)? You can either glue it to the end and hope something else doesn't need to be changed, or you can create a new packet. Both options are bad.

If they used XML for the communication (or JSON or whatever), they could add/remove fields for certain games fairly easily, and modify the protocol for some versions without breaking older clients.

I don't know if they'll do it or not, but I certainly don't think it's "pure stupidity", which is what I was originally responding to.

[Camel]

That's a pretty poor argument for XML, iago, because you can apply the same logic to a binary protocol: They could make the same packets do different things for different clients, but it would be just as silly as using XML.

The real advantage of XML (or JSON, or whatever) is that order is always irrelevant, which is a function of having named fields. SID_READ/WRITEUSERDATA are simple examples of how how named fields can be used in a binary protocol.

[iago]

You can add/remove fields in a binary protocol, but you have to be very specific about it. With XML, the fields are named so you can easily ignore you don't need and parse fields you do.

[Camel]

Binary protocols can be extensible! There is one particular protocol I want to cite, but the application which it powered has been dead for years, and I can no longer find the protocol doc. The application was called HotLine, and it used an extensible binary protocol; it used application-level packets, and inside of the packets there could be any number of fields which were given type by an INT_16 (types could be "username," "flags," etc). Towards the end of its life (in the last revision of the protocol), they added a packet that allowed an administrator to view and edit *all* of the user accounts that the server maintained - to do this, they took the "read/write a single user" packet, and placed the entire packet's payload inside of a field, enumerated that over every user, and sent it as one huge packet.

The original argument against XML (made by MF) was that it generates a lot of pointless overhead - in addition to that, it takes longer to parse that overhead because it is a variable length string. I'm with you in your argument that an extensible protocol is favorable, but XML would not be my recommendation - the BNCS servers do need to take efficiency wherever they can get it.

[MyndFyre]

I was just discussing this kind of problem with our lead interactive developer at the office.  He asked whether we could use AMF3 instead of XML for our new client's new site, which is built on Flex.  I'm happy to accomodate either way, but he made the point of saying: a 3kb message payload in AMF3 might correspond to a 100kb message payload in XML.  You can't get that kind of bandwidth usage and speed out of XML.

[nslay]

i speculate everything is doing to use an xml-based protocol. it'll all use udp, too. udp without tcp-mimicing features. it'll use another modified SHA-1.

this is pure speculation, though.

While UDP is more convenient to process, its dangerous to use for this kind of communication since it is crucial for messages to be received reliably and in order.  A better alternative to UDP is to use the new SCTP protocol.  SCTP offers UDP+TCP behavior.  SCTP communicates with datagrams and guarantees they are received reliably, in order and in whole.  It introduces a new type of socket SOCK_SEQPACKET.  Though FreeBSD's documentation indicates SCTP has support for SOCK_STREAM (FreeBSD's is the reference implementation).

http://en.wikipedia.org/wiki/SCTP

[nslay]

XML seems like overkill.  Isn't it expensive to process and quite redundant?  I can't remember, but I seem to remember there being security advisories in portaudit for XML parsers (very easy to mess up string processing in C).

I think SCTP + packed structs would be the best way to do it.  Read an entire Battle.net packet in one call to recv().  No stupid receive logic needed to deal with fragmented packets.  No parsers either.

[warz]

oh, ok. can we still use xml though

[Camel]

oh, ok. can we still use xml though

Only if Blizzard decides to use it - but they won't.

[edit] @MF, what's AMF3?

[nslay]

oh, ok. can we still use xml though

What makes XML so appealing for this kind of application?  What kind of load would a battle.net server experience processing XML messages from clients?  XML is also redundant (meaning, there's extra information in the format that contributes nothing to the actual encoded data), what kind of bandwidth demands would there be?  It seems like using XML would significantly increase server load and bandwidth demands.  Also, current clients support a less redundant protocol that requires no parsing...what are motivations for Battle.net to support a brand new protocol instead of using the existing protocol?

[MyndFyre]

oh, ok. can we still use xml though

Only if Blizzard decides to use it - but they won't.

[edit] @MF, what's AMF3?

AMF3 is the new Flex-based Action Message Format by Adobe.  (more)

[warz]

oh, ok. can we still use xml though

What makes XML so appealing for this kind of application?  What kind of load would a battle.net server experience processing XML messages from clients?  XML is also redundant (meaning, there's extra information in the format that contributes nothing to the actual encoded data), what kind of bandwidth demands would there be?  It seems like using XML would significantly increase server load and bandwidth demands.  Also, current clients support a less redundant protocol that requires no parsing...what are motivations for Battle.net to support a brand new protocol instead of using the existing protocol?

cause then i can just be like "wtb server token, or something" and it'll just give it to me? seriously. xml.

[MyndFyre]

cause then i can just be like "wtb server token, or something" and it'll just give it to me? seriously. xml.

Seriously?  Contribute something or shut up.

[warz]

cause then i can just be like "wtb server token, or something" and it'll just give it to me? seriously. xml.

Seriously?  Contribute something or shut up.

Well, you can't be too hard on me. This entire xml discussion was sparked by me not being serious. So maybe I'm good at getting others to contribute without them knowing.

But, no, I would not go with XML for future b.net modifications. There's a lot of places XML can look like a good idea and make sense but it also comes at the potential cost of having to transmit/receive a lot of unnecessary overhead. At Nortel we use something almost identical to b.net's protocol for our LTE mobile broadband protocol. It's more specific to individual bits where as I think b.net's seems to deal with bytes - I'm sure b.net's does this too, though. It's pretty cool that an informative packet can be like 3 bytes long. It's obvious XML would not be something we'd use here, though.

I'd think XML would be more or less for storing data and then reading that data - and using some other protocol to xmit. (if it's a lot of data)

[nslay]

cause then i can just be like "wtb server token, or something" and it'll just give it to me? seriously. xml.

Seriously?  Contribute something or shut up.

Well, you can't be too hard on me. This entire xml discussion was sparked by me not being serious. So maybe I'm good at getting others to contribute without them knowing.

But, no, I would not go with XML for future b.net modifications. There's a lot of places XML can look like a good idea and make sense but it also comes at the potential cost of having to transmit/receive a lot of unnecessary overhead. At Nortel we use something almost identical to b.net's protocol for our LTE mobile broadband protocol. It's more specific to individual bits where as I think b.net's seems to deal with bytes - I'm sure b.net's does this too, though. It's pretty cool that an informative packet can be like 3 bytes long. It's obvious XML would not be something we'd use here, though.

I'd think XML would be more or less for storing data and then reading that data - and using some other protocol to xmit. (if it's a lot of data)

I thought the goal of a troll was to provoke anger in a discussion by carefully crafting a point that is controversial and flaw filled as to entice response.  I don't think anybody is religious enough over battle.net protocols to be angered by your comments.