Moak's protocoll v0.7 proposal (REVISED, see rev 1 posting below, click here)

* Take v0.6 handshaking and leave the 3rd step away ([1]).
* Replace GIV with a real HTTP style PUT/POST header
* Document GUID tagging
* Rename Ultrapeers back to 'Superpeers' or 'Superservants'
* Use ISO Latin 1 character set (ISO 8859-1) in search queries/queryhits/HTTP filenames.
* Think about an Unicode addon, for world wide file trade (including Asia)!

Hope you like it. Comments, more ideas? Moak

PS: Okay some explanations are needed. Take a look into current 0.6 handshaking in [1] [2]. The client initates a connection together with HTTP-style headers, the other side answers with an OK and more HTTP-style headers. So far so good (great idea and very flexible)! Then there is this 3rd step (before sending binary data stream), unnecessary I think: the client sends another OK. I see no purpose for the 3rd step, kick it?

Here is a sample interaction between a client and a server with a new v0.7 handshaking:
All information are exchanged in a two step handshaking.

In the very rare case the servants need to shake hands once more and exchange more information (does it ever happen?), they can just continue with another "GNUTELLA/0.7 200 OK" round, but they don't need to. Suprised? This works because the binary message stream will never start with "GNUTELLA" (the first 16 bytes will be the GUID in a descriptor header, which byte 9 is allways 0xFF following the modern GUID-tagging rules, pseudo code: guid[8]=0xFF). (Revised, see rev 1 posting below)

Next point, replacing GIV with PUT or POST, see [3] [4]. Should be self explanary, the HTTP protocoll [5] allready provides a "upload" functionality... we can use this instead of cooking our own HTTP. For sure Gnutella's Push descriptor is still needed, no change here.

Next point, document GUID tagging as mentioned in [6]. New developers need to have a full documented protocoll to contribute.

Next point, umm, Giga-, Hyper- and Ultrapeers [7]. Can we just forget about marketing and name them simply 'Superpeers' or 'Superservants' please. :-) Only the name 'Supernode' is trademarked (Kazaa/Morpheus/Grokster use only this term on their webpages, eDonkey uses the term 'server'), everything else is free. AFAIK good old Clip2 was first mentioning superpeers or refelectors.

Last point, character set. I think we still have none and Gnutella is world wide, eeks! In a first step I suggest to orientate at the web and use ISO Latin 1 (ISO 8859-1) at least, US_ASCII or "wild wild west" isn't enough [8]. Okay, ISO Latin 1 is very selfish, I live in Europe and this will fit my needs. We also should think about a Unicode alternative, perhaps together with file hashs this will perfectly fit into a new HUGE proposal [9].

PPS: Thx to Tamama and Mike Green (Emixode), #gnutelladev

[1] Gnutella v0.6 Handshake Summary - http://www.gnucleus.com/research/connect.html
[2] Gnutella v0.6 Handshake - http://groups.yahoo.com/group/the_gd...ing%20Protocol (Yahoo account required)
[3] Gnutella V0.4/0.6 File Transfer Summary - http://www.gnucleus.com/research/transfer.html
[4] Gnutella protocoll specification v0.4 revision 1.2 - http://www.clip2.com/GnutellaProtocol04.pdf
[5] HTTP/1.0, RFC 1945 - http://www1.ics.uci.edu/pub/ietf/http/rfc1945.html
[6] Gnutella GUID tagging - http://groups.yahoo.com/group/the_gdf/message/1397
[7] "Ultra"peers - http://groups.yahoo.com/group/the_gd...ltrapeers.html (Yahoo account required)
[8] ISO 8859-1 character set - http://www.htmlhelp.com/reference/charset/
[9] "HUGE" - http://groups.yahoo.com/group/the_gd...roposals/HUGE/ (Yahoo account required)

Without the 3rd step in connecting to a server additional requests are very hard(read, non-conform.. like the HTTP/1.0 and GNUTELLA/0.4 on the same port.) to do. Thus one could see this as a 'future guaranty' that extensions could be implemented. However what would a server do if it would get a bad(read, unknown) header? Right!, usually they disconnect. This needs to be more documented.

However you most likely do not need the additional requests because in the two headers that are send to and from the server is all the information one would probably ever need. If one could think of a request that needs to be done outside the initial 2 handshake headers, please post them here


The Uni code character sets could be negotiated in the header. However if not all clients support unicode then searches might come out wrong (or worse, block) at a given server in the network. Also unicode is 2 bytes, so it will add extra overhead on the network.

Example:

A doesnt do unicode
B does
C does

A - B - C

B needs to 'downgrade' search requests and 'upgrade' searchhits from A to B or C. This would be very hard for example chinese to standard ASCII.

Some more thought might be usefull, but I like the idea


I didnt even know about the 0xff at byte 9. It serves no real long time purpose either. New clients will simply use the new standard, and they can be just as bad at routing (read, none) and ping/pongs as the old clients. Some non-yahoo group (emailed) based paper on these GUIDs would be nice indeed.

Tam

Hi Tamama!

> Without the 3rd step in connecting to a server additional
> requests are very hard [...] to do.

Hmm, really? You can just repeat the first steps again and again (if you really need it, 4 steps, 6 steps, 8 steps...). So we are flexible here. In most cases two steps will be fine, I guess. The transition from connect-sequence (HTTP-headers) to binary messages (Gnutella descriptors) could be easily detected: If there is no further string "GNUTELLA/", the binary data stream starts and will run until disconnection of that peer. (Revised, see rev 1 posting below)

> However what would a server do if it would get a bad [...] header?
> Right!, usually they disconnect.

They ignore it as far as I understood the v0.6 handshaking? Each side tells what it supports or wants (2 sides = two steps), then they start to talk Gnutella. If one side doesn't know a feature of the other side, it ignores it. When both sides supports a feature (e.g. 'Query Routing'), then they can use this feature.
It's similar to what happens when a v0.4 client connects a v0.6 client. The old client will connect with "GNUTELLA CONNECT/0.4<lf><lf>", the modern client will recognize this and not continue with a 0.6 connect sequence, instead use the "common language".

> If one could think of a request that needs to be done outside
> the initial 2 handshake headers, please post them here

Dito

> B needs to 'downgrade' search requests and 'upgrade' searchhits from A to B or C.
> This would be very hard for example chinese to standard ASCII.

Yep, veeeery hard! *g* Therfore I suggested to use only ISO LATIN 1 as a default, no negotiation in the connection header!
But within Query/Queryhits a asian client might want to use Unicode (e.g. chinese characters). All other asian clients will understand and could reply... non asian clients don't understand (they don't have asian files), but can route without problems. This type of communication does perfectly fit into the HUGE proposal. With a unicode standard all world wide servants can communicate and speak the same language. Actually also non asian clients could be part of a world wide swarming, when they understand UNICODE.
I also prefer ISO_LATIN 1 as default, because it is a one byte code per character, while UNICODE is a multibyte code! UNICODE as default would unnecessarily increase the allready high broadcast traffic.

> Some non-yahoo group (emailed) based paper on these GUIDs would be nice indeed.

Hehe, yep. At least the required Yahoo account is very anoying IMHO and doesn't help to attract new developers.

Greets, Moak

ok after some talking etc etc

[] = optional
--- = seperator, in between lines are \r\n seperated
... = repeat

client:

---
CONNECT GNUTELLA/0.7
User-Agent: FastFinder 3.4
ultrapeer: 0.1, 0.2
Gnutella: 0.7
[more header-information]

---

server:

---
GNUTELLA/0.7 OK All dandy and fine nice mister client
User-Agent: Peerhana 0.5
ultrapeer: 0.2
ultrapeer: 0.3
Gnutella: 0.7, 0.8, 0.9
[more header-information]

---
<binary data starts>

As you can see header fields are used to tag _all_ allowed extensions. The extensions that are common are automatically activated.

header fields are build up as the following:

&lt;name&gt; : &lt;content&gt;
&lt;name&gt; = &lt;string&gt;
&lt;content&gt; = &lt;string [,&lt;string&gt;...]
&lt;string&gt; = array of characters not containing special characters like \r\n : , etc.

This allows for shorter headers as version numbers can be put on 1 line


Tam

Hi, talking with Tamama, Emixode and Bmk I found it necesarry to overwork the proposal a little bit. :-)

Main goals for a new 0.7 protocol are:
- full HTTP alike connections, to achieve an easy parser
- straight and simple connection sheme, easy and fast
- very flexible desgin for later needs, avoiding a 0.8 in near future *g*
- don't hurt old v0.4 servants, let them still be operable

Appendix A: Extended HTTP style connecting
First of all we make the connection sequence full HTTP like (to use the same parser on all incoming connections). I suggest to use 'CONNECT REQUEST GNUTELLA/0.7', the response 'GNUTELLA/0.7 200 OK' is allready correct.

Then I suggest a slightly different connection handshake to what I wrote in the original post. A v0.7 servant will implement too alternatives:

* 1st handshake alternative: Fast-Connect
Overview: Default handshake, fixed 2 steps (may be repeated in case of error codes)

Here is a sample interaction:
The Fast-Connect is default, it will be the most used connection type. Easy and simple. Each client tells usually _all_ (interesting) features it supports in the headers, once. Unknown headers are ignored, for known headers the from both highest supported version is used. After those two steps (two clients = two steps) the binary messages start automatically.
Two exceptions: a) If the server does not respond with '200 OK' the same procedure is started again (perhaps the server disconnects, but it doesn't need to). b) If the client or the server sends an Header 'Full-Handshake: Yes', they _have_ to continue with the full handshake (described next).

* 2nd handshake alternative: Full-Handshake
Overview: Optional, minimum 4 steps (+ 2*x steps)

Here is a sample interaction:
The Full-Handshake is similar to number 1, optional, whenever one side needs/requests it with the 'Full-Handshake'-header (see comments above). The first 2 steps are the same as in number 1, but this alternative requires an additional 2 step ending handshake before beginning the binary messages (= 4 steps minimum). In the example above you will see two more handshakes steps in between (= 6 steps), even more are possible. The ending handshake is needed as a seperator/transition from HTTP handshaking to binary data, because this alternative allows more then 2 fixed steps. In contrast to the original v0.7 proposal, this is less "hacky". :-)

Why doing all this? The whole connection sheme was designed for easy parsing + effectivity + flexibility. The technical background: While we mostly will use the first intuitive two-step handshake, we introduced the optional flexible handshake for future/extended use (currently we do not need it, especially if someone wants to transfar proprietary data then let those clients deal with it, proprietary is nothing that a handshake should care about, better improve the protocol for the sake of everyone). Especially the second alternative may look oversized, but after long discussions I think this is the way to fit all needs and it's highly HTTP orientated (HTTP we allready use and it promises to be a robust and successfull design). Developers, plz play arround with all posibilities you could imagine, this design should hopefully make your code easy & flexible.

Summary: A V0.7 client will understand the following HTTP and GNUTELLA methods: GET, PUT, CONNECT, START (and GIV, GNUTELLA).

GET for downloading files - e.g. 'GET /get/1283/gnutti.deb HTTP/1.0'
PUT for uploading files (was GIV in v0.4) - e.g. 'PUT 1283:72814A49E69D0F43AAB400/gnutti.deb HTTP/1.0'
GIV for backwards compatibility only, see PUT - e.g. 'GIV 1283:72814A49E69D0F43AAB400/gnutti.deb HTTP/1.0'
CONNECT to initiate a connection handshake - e.g. 'CONNECT REQUEST GNUTELLA/0.7'
CONNECT to continue a started connection handshake - e.g. 'CONNECT HANDSHAKE GNUTELLA/0.7'
GNUTELLA for backwards compatibility only, see CONNECT - e.g. 'GNUTELLA CONNECT/0.4'
START to end the handshake and start the binary data stream - e.g. 'START BINARY GNUTELLA/0.7'

Notes: Do not continue an established handshake with CONNECT REQUEST (this will reset headers and start over), use CONNECT HANDSHAKE. You should not add proprietary data to a Fast-Connect, it would be ignored by all other clients and wastes bandwith, instead take a look at the Full-Handshake example.

Some geek notes: Our Gnutella specification consists of HTTP 1.0 methods (GET, PUT) and GNUTELLA specific methods (e.g. CONNECT, START), all are full HTTP like. Our Gnutella specific methods allow multiple headers, so they actually look more like a HTTP 1.1 request. We define 'Content-Length=0' if the Conten-Length header is not given in Gnutella specific methods (Tamama *g*). This avoids adding 'Content-Length=0' to every CONNECT|START request, since we have no content/payload here.

Response codes are typcial HTTP like responses including status code - e.g. 'GNUTELLA/0.7 200 OK'. [1]

Appendix B: New GUID-Tagging
We introduce a new GUID-Tagging style, first the documentation then additional explanations:

* Byte 16 is used to show the highest supported protocol version, pseudo code: guid[15]=0x07 in this protocol version (up to v25.5 is possible with 8 bits, every value below 4 is treated as v0.4).

* Byte 9 is used to sign if the peer supports special features, bitcoded, low active, 0xFF means no features (backwards compatibility with older clients). The coded features in those bits are important for routing/broadcasting messages:
Bit 0: Peer acts as superpeer (important for network structure)
Bit 1: Peer acts as proxy/tunneling peer (important for firewalled or NAT-routed hosts)
Bit 2: Peer does understand and request metadata
Bit 3: Peer does understand and request file hash
Bit 4: Peer does understand and use UNICODE (UTF-8 encoded)
Bit 5: Peer does have chat support (important for community idea)
Bit 6-7: reserved
The 2 higest bits are reserved/locked and should be allways 1! Again backwards compatibilty, very old clients had a 10XXXXXX setting, which we have to avoid. The default for current v0.4/v0.6 clients is 11111111 which is also our default = no features.

Why using (oldfashioned) GUID tagging over Handshaking? No, it's a new idea behind and both work together. GUID-features are routed along the network (tell status features a far away host), while handshaking features can only used with the direct connected hosts. You can shake hands with direct connected hosts, but not with a hop=5 away peer.
Mainly the GUID feature bits are used to avoid unnecessarily routed/broadcasted messages, e.g. only when a peer understands metadata you need to send him metadata. When a peer just searches for alternative download locations (searching with a hash), the other peers should not answer with metadatas to save bandwith. Secondarily GUID protocol version provides a faster connect. A host cache will receive a PING from any host connecting, it will answer with collected PONGs. [2] Those PONGs will now carry the Gnutella protocol version of the servant inside the GUID. E.g. you can now directly connect an old 0.4 servant with the old protocol or only connect to superpeers. Connecting is faster.

Notes: Feature bits should be discussed, before finishing this protocol version. To detect client version and features use the following pseudo code (this is NO 100% guarantee!):

int protocol = 4; // fallback to v0.4
struct {
unsigned reserved : 2;
unsigned bChat : 1;
unsigned bUnicode : 1;
unsigned bFileHash : 1;
unsigned bMetadata : 1;
unsigned bProxyTunnel : 1;
unsigned bSuperpeer : 1;
} features = 0; //no features yet
if((guid[8] & 0xC0)=0x80) protocol=4;
else if(guid[15]>=0x04) protocol=guid[15];
if(protocol >= 7) features = guid[8] ^ 0xFF;

Appendix C: Character set and UNICODE expansion
While talking with some other developers, I found out that my idea was too short/confusing, here is a more detailed suggestion:
* First use ISO_LATIN 1 as basic charset in all HTTP header and/or Gnutella strings (propably you allready did this). This won't hurt anyone and will fit most European/American/Australien needs to transport national special characters (e.g äåæ). But... and this is a big "but", this character set will not fit asian, russian needs and more languages. While Gnutella messages are routed through various servants it is impossible to translate one chracter set into another, the solution is UNICODE.
* Second use optional UNICODE in Gnutella strings (Query/Queryhits), if you really need it. Unfourtunately Unicode is a two byte code per character + Unicode contains zero bytes (A = 'a', '\0') which is a problem too. So Unicode must be encoded, we use UTF-8 (see later post for details).

Why not using Unicode allways? Oops, the binary datastream would not be backwards compatible with v0.4 and it would also blow up Query/Queryhit unnecessarily compared to ISO_LATIN 1 (1 byte vs multibyte characters, bigger size = bigger query traffic, eeks). Why not using another character set than ISO_LATIN 1? Whatever you choose it wouldn't fit a world wide need. Do we need to negotiate UNICODE in the connect handshake? No, since UNICODE is routed in Query/Queryhits it will travel the whole network mesh and is not affected by direct connected hosts. Instead we use the GUI-tagging (see above).

Notes: The two step solution above is my suggestion, perhaps is is better to skip Latin1 completly and use UTF-8 only (topic of further investgations). With the two step solutions all clients speak a common character set (ISO_LATIN 1), very easy to implement, the UNICODE (UTF-8) is optional for clients which want to offer an extended language support or do not use ISO_LATIN1 codepage on their system. While UNICODE (UTF-8) is encoded inside the Query/Queryhit every "basic" client will route it without any problem. Okay, an old client will not understand it, but that is no problem, e.g. a non asian client will usually have no asian files.
I might post a new proposal for a "specialized horizons" header. This allows to group similar interests and traffic together + keep in touch world wide. In this special context here it will help to route e.g. asian unicode primary between ppl offering asian content.

Appendix D: Backwards/Upwards compatibilty
Backwards compatibility with incoming protocol v0.4 is full guranteed. All incoming v0.4 clients should be answered with the simple v0.4 handshaking. Outgoing connections should be started depending on the GUID tagging, however if v0.7 doesn't suceed try v0.4. About v0.6 backwards compatibilty, while the handshake is incompatible with v0.7, I suggest to make this decision client vendor dependent, if you wanna support v0.6 just do it, it's optional.

Here is a sample handshake, old client knocking at the door:

Upwards compatibility is also supported (as long as the handshake is compatible, hopefully it is).
Here is a sample handshake, newer client tries to connect:

Appendix E: Misc
More stuff for brainstorming and documentation....

Let's collect and write down everything that was introduced since v0.4 and is fact in modern clients! I think about blocking webbrowsers (with Referrer header), what more?

Let's think about a chat specification. Perhaps IRC based or IRC DCC for inbetween client communication, perhaps HTTP based.

Integrate the Bye-Descriptor [3] into the v0.7 protocol. Make this message also full HTTP (pure text string, no byte codes) [1]. Perhaps we should also notify downloader and send them a new HTTP BYE message?

Define clients research and retry behaviour. When a peer drops the connection, you want to resume/reconnect. Such a retry should be well defined to avoid "hammering" foreign IPs.

Most important points at the end, a big TODO, workout and define protocol features for: superpeers, dynamic traffic routing, hashs, metadata and (friendly) anti-freeloading behaviour (we had various posts in this forum discussing those topics).

Ah, don't forget to vote for 'Superpeer'... not Ultra-Hyper-Giga-Peer. :-) Keep Marketing and prorprietary ideas out of Gnutella!

Comments, more ideas? Greets, Moak

PS: I might summarize all ideas into one paper later, we still need to work out this proposal. Feedback is highly welcome! Send me a PM (private message) or meet me on IRC. :-)

[1] HTTP/1.0 Status Codes, RFC 1945 - http://www1.ics.uci.edu/pub/ietf/htt...l#Status-Codes
[2] Gnutella Host Caches - http://www.gnutellaforums.com/showth...&threadid=5807
[3] Bye Descriptor - http://groups.yahoo.com/group/the_gd...ls/BYE/Bye.txt (Yahoo account required)

I've got a couple of questions:

1) Why won't you use HTTP 1.1 (it has the one or the other advantage)?

2) Does the gnutella 0.4 spec say anything about the charset to be used for gnutella messages? I already searched for files with äöü... and also found them with LimeWire.

3) Wie heißt die Transformers Episode mit dem Hasstaub auf englisch?

1) We do not need it? Actually the new v0.7 Gnutella specific methods look more HTTP/1.1.

2) Protocoll v0.4 mentions no character set, they speak about "ASCII encoded" strings (= wild wild west, every chat above '\127' is undefined yet). Protocoll v0.7 defines ISO_LATIN 1, which includes US_ASCII as subset (backwards compatible).

3) Transformers Volume 7: The Return Of Optimus Prime

Appendix F: LAN Auto-find & Proxy configuration

How about an "auto-find" feature for servents running in a LAN?

The idea: Multiple servents running in a LAN could easily find each other and connect automatically. Perhaps on a LAN party or in a home network, servents will find each other without running an additional hostcache or manual configuration (plug'n'play). Similar to multiplayer games where you click on "LAN games" and then you'll see all running servers, no need to configure IP/port manually.

Implementation: Each servent sends a UDP broadcast [1] on startup to the LAN (only internal LAN devices, not to the internet) and every servent will answer with an UDP "pong". The binary data within those UDP datagrams should be identical with Gnutella's ping/pong descriptor (for reusing the existing methods/code!).

There's more... supplementary idea 2: It may usefull to define a servent which is the link to another network segment or to the internet, a superpeer with proxy/tunneling ability. All clients in the LAN will auto find this servent (LAN UDP broadcast again) and then use this superpeer all together to connect to the internet, instead of each and everyone connect on their own.
I think this last feature is very important for two reasons: a) Internet bandwith is used more efficient and b) a superpeer proxy could increase download succes for people behind a NAT router. A NAT router usually allows no incoming connections, peers behind two different NAT routed networks can not share files (also not via Gnutella's push, see here). Such hosts are the majority following Limewire's host statistics [2]. A superpeer proxy in the LAN could be the missing link and reroute/tunnel those requests to the correct servent = more shared files, higher download rate.

Implementation: Similar to UDP broadcasts above, a proxy servent will set the equivalent GUID features bits (bSuperpeer, bProxyTunnel) before answering with an UDP "pong". Additional handshaking headers are needed to negotiate proxying and tunelling [3] between clients and superpeer, topic for further investigations.

Notes: Do also cross check with the 'Your-IP' handshake header in Appendix A, this header is used to tell the local peer (or tunneling proxy peer) the external IP of your NAT router (IPs could be dynamic since most ppl do not have a static IP). Furthermore see Apendix B, the GUID tagging Byte 9 Bits 0-1 define the mentioned superpeer and proxy/tunelling ability.

Result: We achieve a full automatic LAN client find + automatic proxy configuration (elimination of today existing bottlenecks with NAT routers, traffic minimization).

[1] UDP Broadcasts - http://tangentsoft.net/wskfaq/interm...html#broadcast
[2] Limewires's Gnutella Network Size - http://www.limewire.com/index.jsp/size
[3] Gnutella Tunneling - http://groups.yahoo.com/group/the_gd..._discusion.txt (Yahoo account required)

"* Take v0.6 handshaking and leave the 3rd step away (see [1]). (Revised, see rev 1 posting below)
* Replace GIV with a real HTTP style PUT/POST header
* Document GUID tagging
* Rename Ultrapeers back to 'Superpeers' or 'Superservants'
* Use ISO Latin 1 character set (ISO 8859-1) in search queries/queryhits/HTTP filenames.
* Think about an Unicode addon, for world wide file trade (including Asia)! "

The problem with this is that it's not very backward compatable. Trust me there have been hundreds of these type of proposals, none have made it.

The reason 0.6 made it was the it was backward combatible. It only involved the hadn shaking mechanism, so a servent could have both 0.6 and 0.4 connections.