An attempt to explain the say2 protocol in a less terse way. (last update: 02-sep-2001) C: client sends S: server sends This document describes say2 protocol 0.2 or [0, 2]. *** Why another chat protocol? Short answer: Because nobody's done it right yet. Long answer: In the mid 90s, a bunch of IRC users started discussing ways to improve IRC. The existing network was scaling horribly, and patches were only providing short-term life extensions. We came up with a bunch of good ideas. I won't even try to remember everyone's name who provided suggestions and criticism -- I'll only say that pretty much none of the major concepts of say2 were invented by me. They came directly out of the (sometimes very heated) discussions on the "irc3" mailing list. Since that time, IRC has stagnated, and many new chat networks have arisen. The newest ones drop the concept of a channel completely, and call themselves "IM" networks (examples: AOL, Yahoo). All of these recent chat networks are either hopelessly centralized, or have no concept of privacy, or use secret proprietary protocols. Most networks have all or most of these flaws, in fact. Some of them, when you reverse engineer their protocols, make you want to leave the computer industry and join a monastery. But we had already invented a conceptual solution that would allow privacy, decentralized servers, scalability, and would not only keep the concepts of channels ("rooms" to AOL users) but actually give channel operators local control. So for many years I (and my friends) have watched the new networks throw away or ignore virtually all of these discoveries, and assumed that eventually someone would do it right. Now years have past, and we're tired of waiting. They've had their chance, and they flubbed it. So when Greg told me of a project to rewrite the chat system used by his friends in Atlanta (called "say"), I took advantage of the opportunity to share all our previous ideas and got him psyched up to do a new system, and do it right. (Doing it right doesn't mean building huge bureaucracies. It means putting together something large using simple concepts that can stack up like legos without needing 3 months of discussion about each piece.) We came up with enaml first, and then I started working on the servers while Greg started working on the client. Since then, a bunch of others have joined the project and we're always looking for more. Here's another opinion: well, "noone has done it right" really pertains to robey's opinions. :) and, of course, the ultimate answer to why: robey gets bored. robey gets dangerous when he gets bored. he tends to write chat thingies. i'm trying to figure out when my life deteriorated to the point that i have nothing better to do than sit and watch people discuss obscure chat protcols on a friday night http://www.lag.net/say2/ Concept: ENAML. This is a big concept. Read enaml.txt first. Concept: 'nick' vs 'id'. Most messages from the server that identify a specific client will attach both a 'nick' and an 'id'. The 'nick' is always the current nickname the client is using, while 'id' is the user's authoritative identifier (whatever name the user registered). For example, my 'id' is "Robey", so usually my 'nick' and 'id' are both set to "Robey". Whenever I change my nickname to "RobeyZzz", my 'nick' field will change to "RobeyZzz", but my 'id' field will remain as "Robey". Clients can use this to uniquely identify a registered user (or a single continuous guest session) despite repeated nickname changes. Most commands that take a 'nick' will also take an 'id' for the same reason: to provide a little flexibility in how a client (or the end user) decides to address someone else. Concept: UTF8. UTF8 is a way of encoding Unicode across an 8-bit clean pipe while preserving the lower 128 characters (ASCII). All text sent in the say2 protocol is encoded in UTF8. This means not only the text sent between two clients as private messages or on a channel, but also things like the client version string, taglines, error message explanations -- everything that is intended to be read by a human. Text that is not intended to be read by a human will never use anything other than 7-bit ASCII, so clients should feel free to uniformly treat the entire protocol as being in UTF8. Protocol errors are reported this way: S: error:{ type:"protocol" command? text } command: the client command that caused this response (for example, "message" or "whois") -- not all errors are associated with a specific client command, though text: human-readable text explaining the error But, for some operations, specific error messages may be defined that give more context to the client. (Because in some cases the client may want to know what kind of thing went wrong.) The "protocol" error will only occur due to an error in the client. For a few operations (like sending wall messages), the server may respond with a "not-permitted" message, meaning the user doesn't have permission to perform the operation: S: error:{ type:"not-permitted" command text? } command: the client command that caused this response text: optional human-readable text If an element of a command was too long (like a password or icon), the server may respond with: S: error:{ type:"too-large" command item text? } command: the client command that caused this response item: the element of the command that was too large (ie "icon") text: optional human-readable text *** SERVER GREETINGS S: hello:{ protocol } protocol: list of numbers: [major, minor] describing the say2 protocol version supported by this server S: server-info:{ server version text? uptime? } server: human-readable text describing this server software (example: "SuperServer") version: version string for the server software (example: "1.1") text: human-readable text to be displayed to the user -- this can contain anything at all uptime: epoch time (seconds since 1970) since the server started S: motd:{ text } text: human-readble text to be displayed as a "message of the day" for the server -- sent automatically on initial login The 'hello' message is an initial greeting intended only to confirm that the server connected correctly and is speaking say2 protocol. ** REQUEST SERVER INFO C: server-info S: server-info:{ server version text? uptime? } server: human-readable text describing this server software (example: "SuperServer") version: version string for the server software (example: "1.1") text: human-readable text to be displayed to the user -- this can contain anything at all uptime: epoch time (seconds since 1970) since the server started A client can periodically request the server-info block again by sending an empty 'server-info' command. *** REQUEST MOTD C: motd S: motd:{ text } text: human-readable text to be displayed as a "message of the day" for the server The client can request the MOTD to be repeated in this way. *** LOGIN AS GUEST C: login:{ nick email tagline? client? address? } nick: requested nickname (max 32 chars: first char from [A-Za-z_-], others from [A-Za-z0-9_-]) email: email address (max 40 chars) tagline: human-readable text displayed in /whois but otherwise not important to the server (max 60 chars) client: human-readable text that identifies the client version (example: "say2.pl v0.3") (max 60 chars) address: internet address, in "host:port" format, where other clients may contact this client for private conversations [FIXME: no protocol work has been done on these client-to-client connections yet -- volunteers?] S: welcome:{ nick id } nick: the client's nickname, echoed back id: a unique ID identifying this user Possible user-caused errors: S: bad-nick:{ nick text } nick: the nickname that was unacceptable text: a human-readable reason why *** LOGIN AS REGISTERED USER C: login:{ id password nick? client? address? replicant? } id: the registered nickname password: the user's password nick: optional nickname to use instead of your id (by default, your id is your nickname when you login) client: human-readable text that identifies the client version (example: "say2.pl v0.3") (max 60 chars) address: internet address, in "host:port" format, where other clients may contact this client for private conversations replicant: flag that's present if the client would like to attach to an existing session without disconnecting any other connections S: welcome:{ nick id attached? replicant? host } nick: the client's nickname, echoed back id: the client's registered id, echoed back attached: flag that's present when the connection was attached to a previous session (see below) replicant: flag that's present if the new connection attached to the session without disconnecting any other connections host: hostname the new connection is coming from The tagline is pulled from the user's preferences. If the user was logged in on another socket connection, that connection is broken, and the session resumes on this new connection. The broken connection will receive (right before being disconnected) a 'logout' notice with the 'detached' field set to the hostname of the new session. The new connection will have the flag 'attached' present in its welcome message, to indicate that the session was attached. No other indication is given to the client, and no indication is given to any other client. However, adding the 'replicant' flag to the login command will ask the server not to disconnect any other sockets on this session. If that succeeds, the 'welcome' message will contain the 'replicant' flag also, and all traffic to this session will now be mirrored to all of the session's connections. The new 'welcome' message will be sent to all old connections too, so they can see that a new connection is present (and what host it comes from). If the session already has the maximum number of replicants, a "too-many-replicants" error message will be sent (see below). Whenever a new connection arrives on a session, incoming messages (see "status" below) are turned on. If this is a change in state, a "status" message will be sent. You may specify a nickname to use instead of your registered id, but this will not take effect if you are connecting to an existing login session. Possible user-caused errors: S: bad-address:{ address } address: the address that was unacceptable S: error:{ type:"bad-password" command:"login" id text? } id: the nickname of the attempted login text: optional human-readable text S: error:{ type:"too-many-replicants" command:"login" id text? } id: the nickname of the attempted login text: optional human-readable text *** LOGOUT C: logout:{ } S: logout:{ text? detached? } text: human-readable text, usually meaningless detached: hostname of the new connection that detached you (if present; see 'login' above) This is how a client "politely" disconnects. The server may attach a text message for the user (jenova always sends "Goodbye!"). Typically the server will disconnect immediately afterward. S: replicant-terminated:{ host } host: hostname of the connection that dropped The server sends this message to any remaining connections on a session when one of the connections is dropped. Clients may use this to track how many connections remain on a session. *** PING S: ping:{ } C: ping:{ } The server will periodically send out a 'ping' message to clients, to make sure they're not stoned. Clients should respond by echoing the 'ping' back. Clients that don't respond within a set period of time (usually a matter of minutes) will be disconnected. If there are multiple connections on a session, each connection is pinged separately. *** CATCHING SIGNON/SIGNOFF S: signon:{ nick id time email } nick: nickname of the user who just signed on id: registered id of the user who just signed on time: signon time, in epoch (seconds since 1970) email: user's email address [probably useless] S: signoff:{ nick id time } nick: nickname of the user who just signed off id: registered id of the user who just signed off time: signoff time, in epoch *** CHANGING USER PREFERENCES C: prefs:{ tagline? notify? password? icon? } tagline: human-readable text displayed in /whois but otherwise not important to the server (max 60 chars) notify: ("on" or "off") when notify is on, the client is notified of all logins/logouts on the server (this makes more sense for channels than for the registry server, obviously) [FIXME: archaic; use buddy lists] password: yep, this is how you change your password icon: binary icon data (GUI clients may use this to display a custom icon for the user -- typically a 32x32 color GIF or PNG); some servers may limit the size of this: jenova limits it to 4k or less (and ought to limit it to 1k) S: prefs:{ tagline notify password-changed? icon? } tagline: echoed back as it was saved notify: echoed back as it was saved password-changed: this key is present if the password has just been changed icon: this key is present if you have an icon set (the icon is NOT echoed back) Note that the client doesn't have to send ANY tags in the 'prefs' command, in which case it's just asking to see what the current values are. (Any preferences that aren't specified remain unchanged.) The server never echoes back a password; instead it includes the 'password-changed' tag if the password has just been changed. It is expected that the possible tags for this command will grow. If a guest login attempts to use this command, the server will respond: S: no-user-record because only registered users can have preferences or attributes. *** CHANGING STATUS C: status:{ mode? text? messages? duration? } mode: ("active", "busy", "away", or "idle") the client's current level of activity: largely user-defined what these mean. text: human-readable description of the current user's state (60 chars max) messages: ("on" or "off") whether the client would like to receive private messages. when off, incoming messages are stored as if the user was offline; when turned on, any stored messages are delivered. duration: an optional # of seconds to retroactively set this status level (default is 0). the server subtracts this duration from the current time when setting the 'time' field of the status responses below. the time will be rounded down to the current time or up to the time of the last status change. this is useful for clients marking themselves "idle": they may have noticed the beginning of idleness 10 minutes ago but only want to change status after making sure the user has been idle for 10 minutes. S: status:{ nick id messages time mode text? } nick: nickname of the user id: unique ID for the user messages: ("on" or "off") whether the client is now receiving private messages. if a new connection arrives on this login session, messages will be turned on again, stored messages will be delivered, and a "status" message will be sent. time: time, in epoch seconds, that the status was set (may be offset by 'duration' above) mode: ("active", "busy", "away", "idle" or "offline") whatever the client has set as its mode most recently. "offline" indicates that the user has logged off. text: any human-readable description set by the client Generally a descriptive text is set for the "busy" and "away" states, so that other users are told why this user is busy or away. For example, if a user is set "away", the descriptive text might be "in a meeting". A client may set a descriptive text even when in "active" mode (unlike, for example, IRC), and it's still displayed. (Perhaps the user is here, but also watching TV.) A client can probe its own status by sending an empty 'status' command. When other users change their status mode or text, the server will broadcast the new info in unsolicited 'status' messages. This should probably follow the client's "notify" preference, but currently it doesn't: FIXME. *** CHANGING NICKNAME C: change-nick:{ nick } nick: new chosen nickname S: change-nick:{ id old-nick nick } id: unique identifier for the user old-nick: previous nickname nick: new chosen nickname Note that the 'id' field will remain the same across any nickname changes. A user may always change nicknames to match her id, but can never change to use the nickname of any other user or any other user's id. When other users change their nicknames, the server will broadcast the new info in unsolicited 'change-nick' messages. This should probably follow the client's "notify" preference, but currently it doesn't: FIXME. *** LIST CLIENTS [as of v0.3.1] C: list-clients:{ type grep? verbose? by-id? } type: ("user", "channel" or "all") whether to list only users or channels, or to list both grep: if present, only list users whose nicknames or ids contain the grep text verbose: (a flag) if present, give the long response form by-id: (a flag) if present, list (and grep) users by their ids, not by their nicknames Short response: S: members:{ type list:[ ] } type: ("user", "channel" or "all") echoes what the client requested list: list of nicknames or ids that are online (and matched by the grep) -- channels are always listed by id since they have no nickname Long response: S: begin-response:{ count total response:"client-info" } count: number of messages to follow total: number of answers there were (might be larger than count, if the server thinks the answer would be too much to handle) response: always "user-info" in this case S: client-info:{ ... } There will always be as many of these as were listed in 'count' above. S: end-response:{ response:"client-info" } *** WHOIS USER (user is online) C: whois: { nick|id type? icon? } nick: nickname of the user to lookup (or:) id: registered id of the user to lookup type: "user" (optional: if missing, "user" is assumed) icon: flag present if you want the icon attribute included in the response S: client-info:{ id nick type:"user" email login tagline? mode status? status-time client? icon? guest? messages } id: registered id of the user nick: current nickname of the user (might be the same as the id) type: "user" always for users email: registered email address of the user login: login time, in epoch (seconds since 1970) tagline: the user's chosen human-readable description, if present mode: "active", "idle", "busy", or "away" status: human-readable text set by the client, if present status-time: # of seconds since the mode or status was last changed client: human-readable description of the user's client software icon: binary icon data (if the "icon" flag was present in the request, and the user has an icon set) guest: flag present if the user is just a guest (unregistered) messages: ("on" or "off") whether the client would like to receive private messages. when off, incoming messages are stored as if the user was offline; when turned on, any stored messages are delivered. S: not-present:{ nick|id type } nick: nickname of the whois query (or:) id: registered id of the whois query type: "user" or "channel" The user mode is set by the user manually between "active", "busy", "away", and "idle". Usually the user's status text is a description explaining the mode. (For example, when the mode is "busy", the status text might be "working on the server".) *** WHOIS CHANNEL (channel is online) C: whois:{ id type icon? } id: the channel's registered id type: "channel" icon: flag present if you want the icon attribute included in the response S: client-info:{ id type:"channel" email login tagline? client? address icon? } id: registered id of the channel type: always "channel" for channels email: registered email address of the channel login: login time, in epoch (seconds since 1970) tagline: the channel's chosen human-readable description, if present client: human-readable description of the channel's software address: hostname (or IP) and port, in "hostname:port" format, where the channel is accepting connections icon: binary icon data (if the "icon" flag was present in the request, and the channel has an icon set) *** WHOIS (offline client) C: whois:{ id type? } id: registered id of the user or channel to lookup type: "user" or "channel" (optional: if missing, "user" is assumed) S: client-info:{ offline id type email tagline? last-login last-logout } offline: flag to underline the fact that this client is not currently logged in id: registered id of the user or channel type: "user" or "channel" email: registered email address of the user tagline: the user's chosen human-readable description, if present last-login: epoch time of the last login by this user last-logout: epoch time of the last logout by this user S: no-such-client:{ id type } id: the registered id from the whois query type: "user" or "channel" *** BOOTING OTHER CLIENTS C: boot-user:{ type? nick|id text? } type: "user" or "channel" (defaults to "user") nick: nickname of the user to boot (or:) id: registered id of the user to boot text: optional human-readable "reason" to display to the channel S: boot-user:{ nick id target-nick target-id text? } nick: nickname of the user who did the boot id: registered id of the user who did the boot target-nick: nickname of the user kicked off the channel target-id: registered id of the user kicked off the channel text: human-readable "reason" given by the user who did the boot This is a weird jenova-ism. THIS WILL NOT BE THE COMMAND USED TO KICK USERS FROM CHANNELS. It's meant as a mechanism for removing users or channels from a say2 registry server. The server may respond 'not-present' (see above) if the target user is not online. FIXME: The response has no "type" field and is broadcast to all users. This is horribly wrong. FIXME: The error response for insufficient access should be a specific response, not just the generic "error" response (so that clients can catch this error and display it differently). *** ATTRIBUTES Every user and channel has a field of optional "attributes": name/value pairs, where the name is an enaml key, and the value is any valid enaml value (string, list, or block -- note that a flag is not allowed). These attributes are part of the permanent record for the user/channel, so they're saved by the server, and the server may set an arbitrary limit on how much data can be stored this way (jenova limits you to 1k). C: get-attributes:{ id type? attribute-list? } id: registered id of the user or channel to lookup type: "user" or "channel" (optional: if missing, "user" is assumed) attribute-list: optional list of attribute names (only those attributes will be displayed); if missing, all attributes are displayed S: client-attributes:{ id type attributes } id: registered id of the user or channel type: "user" or "channel" attributes: block of attributes, consisting of name/value pairs, where the value can be a string, list, or block A client may set its own attributes: C: set-attributes:{ ... } S: set-attributes:{ attributes } attributes: list of attribute names that were added When using 'set-attributes', all name/value pairs inside the block are attributes to be changed. You can set the value to be any valid enaml value except a flag (ie, string, list, or block). Setting an existing attribute overwrites any previous value it had. Listing an attribute as a flag (with no value) causes that attribute to be removed. (Yes, that's kinda hacky.) The server may respond with 'not-present' (see above) to 'get-attributes' if the requested client isn't online. The server may respond with 'no-user-record' (see above) to 'set-attributes' for guest logins. *** LOGIN AS CHANNEL C: login:{ channel password client? address } channel: the registered channel name password: the channel's password client: human-readable text that identifies the client version (example: "superclient v0.4") (max 60 chars) address: "host:port" address that clients can use to connect to the channel S: welcome:{ channel } nick: the channel's registered name, echoed back *** SAY FIXME: The 'say' command only makes sense on a channel, so eventually only channel servers will send or receive 'say' messages. 'message' will still be valid on both channel servers and the registry server. C: say:{ text|act? ... } text: a text message from the user to the channel act: a text action from the user to the channel S: say:{ nick id time text|act? ... } nick: nickname of the user who's talking id: registered id of the user who's talking time: time, in epoch seconds, that the message was sent text: a human-readable text message to the channel act: a human-readable text action to the channel A text message or action is just whatever the user typed at the keyboard. An action is usually displayed immediately after the user's nickname (following an old IRC tradition): for example, and action might be: say:{ id:"Robey" nick:"Robey" time:990898977 act:"sits down" } and might be displayed this way: * Robey sits down The human-readable text in a 'say' command may have CTCP2 formatting codes in it. [FIXME: Add pointer to CTCP2 docs] These can be ignored by removing any text between two control-F's. The 'say' command is unusual in that almost any tags can be present in it. You aren't just restricted to "text" and "act": conceptually, any type of message can be broadcast to the channel this way, just by convention. The server will throw away any tags that conflict with tags it plans to add (nick, id, time) -- all the rest can be specified by the client. This gives you a huge amount of flexibility in making up your own message types if you want. In reality, though, most clients will probably just drop any messages that don't contain a "text" or "act" tag. *** MESSAGE C: message:{ nick|id text|act? ... } nick: nickname of the planned recipient of the message id: registered id of the planned recipient of the message text: a private text message act: a private text action S: message:{ nick id to-nick to-id time stored? text|act? ... } nick: nickname of the user who's talking id: registered id of the user who's talking to-nick: nickname of the recipient user to-id: registered id of the recipient user time: time, in epoch seconds, that the message was sent stored: (flag) present if the message was stored offline text: a human-readable private text message act: a human-readable private text action You will only receive a 'message' if you are the recipient of it, or you sent it. (Receiving the message if you sent it is useful for noting the timestamp and possibly identifying lag. It also maintains consistency with 'say'.) If you are the sender and the recipient (you sent a message to yourself), you will only receive one copy. The fields of 'message' are just like 'say' in that besides "text" and "act", there could be almost any tag included, and that the "text" and "act" values may have CTCP2 format codes in them. If you send a message to a user by his or her id, and that user is not online, the server will store the message for later delivery, and the copy echoed back to the sender will contain the "stored" flag. (That's your indication that the message was not yet delivered.) When the recipient logs on next, all pending messages will be delivered, and all will have the "stored" flag set. The server may respond to a message with 'not-present' (see above) if the target client is not online, and was specified by nickname. It may also respond with 'no-such-client' if the target was specified by id, and no such id exists. *** WALL C: wall:{ text } text: a human-readable text message to be sent to every user online S: wall:{ nick id time text } nick: nickname of the user who's talking id: registered id of the user who's talking time: time, in epoch seconds, that the message was sent text: a human-readable text message to every user online Only the server administrators are expected to have the ability to send a "wall" message. These are messages sent to every user online, and are generally meant to be notifications (of server maintenance, for example). *** HISTORY C: history:{ max? } max: maximum # of history lines to dump (the default is server- specified; jenova defaults to 200) S: begin-response:{ count total response:"say" } count: number of messages to follow (limited by 'max') total: number of messages in the history (may be larger than 'count' if there are more than 'max' messages) response: always "say" S: say:{ ... } S: end-response:{ response:"say" } *** SERVER CERTS C: server-cert S: server-cert:{ host alias:[...]? cert } host: authoritative hostname for this server alias: optional list of hostnames that are also valid for this server cert: X509 certificate with the CA bit (ie. a self-signed SSL cert) C: start-tls S: start-tls This is the simplest way to begin using SSL. After receiving the server's 'start-tls' message, the client should immediately switch into using SSL to encrypt the connection, before sending any further data across the socket. If the server is unable to do SSL, it will respond: S: error:{ type:"no-ssl" text? } text: optional human-readable text [Explain why you want a server's cert.]