My essay Why yEnc is bad for Usenet raised the idea that what we need is "a better way to post binaries on Usenet." However, that idea itself raises the question, "what is a better way?" My answer is in these pages.

Because of the mess caused by yEnc, I am going to say this right up front: This is a work in progress, not a finished standard. Do not implement these ideas in production software! Whatever standard may emerge is virtually guaranteed not to be exactly what is on this page, so please don't turn all of Usenet into beta-testers like yEnc did. If you like the ideas presented here, please come to news.software.nntp and join the discussion. Feel free to write test implementations, but please don't encourage people to use this stuff on Usenet in general at this point in time.

To define a better way to post binaries, there are three areas we must address. My goal is to do this entirely within the existing MIME specification, rather than requiring changes to the spec first. Basing the design upon needed changes to the spec would, at the very least, make this a much longer process, and may doom it to failure.

Due to the nature of MIME, we can look at each of the three areas individually. In addition, I have a fourth section, giving a specific example of how the basic ideas can be extended to add media-specific additional features.

Transfer encoding

The main selling point of yEnc is its smaller encoding. The bloat of uuencode and base64 is unnecessary on Usenet, where the channels of transmission are very nearly 8-bit clean. By defining a smaller encoding within MIME, we can take advantage of this, without giving up all the goodies MIME makes possible.

To take advantage of existing code and gain full backwards compatibility with existing newsreaders, we can define yEnc as a MIME transfer-encoding.

Binary posting

Binaries (single-part ones, at least) can be posted using MIME right now. However, the process could benefit from a more detailed specification. If posting agents were to adopt a consistent set of methods, the entire process could be made faster and easier for the users, and additional features could easily be added on top of the basic specification.

In this section, I present a proposed set of guidelines for how clients should post binaries to Usenet. This includes several small additions to MIME (but is mainly a description of how to work within the current specification), and serves as a basis we can then extend to support multipart binaries as well as other added features.

Multipart binaries

When files are too large to post in a single article, they are split up and posted in segments. In this section, I propose a new MIME content type for transmission of files in multiple segments. The goals are to enable reliable and automatic reassembly of files; automatic searching for and identification of missing file segments; partial reposts, even by third parties; automatic identification and use of later reposts (including partial reposts) even under different filenames; automatic location of metadata such as textual descriptions of files; and faster scanning of newsgroups for items of interest.

Posting images

Finally, I present a detailed example of the kind of media-specific features that can be added to the basic posting specification to make it even more useful. This is a proposed specification for posting images to Usenet which enables clients to show thumbnails of the pictures available in a newsgroup, and allow the user to retrieve the full pictures from them.


NEW: here's a note about the current status of this project.

I have created these specifications in the hope that, if they don't suck, they will provide some benefit to Usenet as a whole. I place no proprietary restrictions on the use of the ideas, methods, or algorithms here, or anything silly like that.

There is sample code available implementing yEnc as a MIME transfer encoding. See the encoding page.

For another take on this issue, see Claus Färber's Large Binary posts on Usenet page.