Why misplaced binaries suck

by Jeremy Nixon, 23 November 2002

There is a long-standing rule on Usenet that binaries may only be posted in appropriate binary newsgroups, and the rest of the groups are for text posts only. In recent years, Usenet has gained a lot of new users and widespread commercially-available access, and many people are either not aware of the reasons for this rule or think it is no longer applicable. However, there are good reasons for it, and it most certainly is still applicable, even more so than in the past.

What the heck is a “binary,” anyway?

As a term of art, on Usenet a “binary” is a file other than plain text, which isn't human-readable. These are often referred to as “attachments.” Pictures, MP3 files, movies, PDFs, and software programs are some types of binaries.

Although HTML is unwelcome in almost all newsgroups, it is not considered a binary. Cryptographic signatures, although technically binaries, are usually considered small enough not to run afoul of the “no binaries” rule, provided they are indeed very small, as a special exception.

A binary file must be encoded using something like uuencode or yEnc in order to be posted to Usenet. Encoding a binary file into text does not make it a text file.

What's the big deal, if you don't want them don't download them!

First of all, don't assume that everyone reads Usenet the same way you do. You may use a program such as Outlook Express, where you can view a list of articles in a newsgroup and only download the ones you're interested in. Not everyone does it this way. Most of the world does not have broadband Internet connections, and many people still pay per-minute or per-hour to access the net; others want to read offline on a laptop, or simply prefer offline reading methods, in which case they use software which downloads the entire contents of a newsgroup for later perusal. Since reading a group this way is much faster than online reading, many people simply use this kind of reading software by preference.

Most people on Usenet use the text groups, not the binary groups. But the binaries use the overwhelming majority of Usenet's resources. The binary-downloading newcomers need only to keep their posts out of the text groups for everyone to just get along.

But it's not about downloading. It's about feeding. Posting to Usenet is not at all like posting to a web-based message board or a weblog. When you post to a newsgroup, your post is automatically sent around the world to tens of thousands of news servers. The full newsfeed, including the binaries, is exceeding 500 gigabytes of data per day, and requires at least a T3 Internet connection (which costs more per month than many people earn in a year) to receive. Most of Usenet's systems are simply not equipped to handle the binary traffic.

In order to enable small systems to continue to exist, they must be able to receive the text groups without being flooded with binaries.

But a server can just filter the binaries it doesn't want.

Due to the way news servers work, filtering the binary posts won't solve the problem. A server has to receive the entire article before it decides whether to drop it, and at that point, the bandwidth is used. If binaries are posted all over Usenet in any newsgroup, only the largest systems would be able to stay on the network. This would eliminate almost every server from Usenet.

There is nothing in the Usenet transit protocol which enables a server to know that an article contains binary data until it has completely received the article. It is not possible to effectively filter binaries to save bandwidth at the receiving end of a newsfeed.

Why should binary groups need to be in special hierarchies?

Newsgroups are named in hierarchies in order to make distributed administration possible. You might think that a server can simply choose not to receive “binary” groups, but it's not that simple. There is no way to inform the world that a particular newsgroup carries binary traffic. So, what we do is designate sub-hierarchies where binaries may be posted.

This arrangement allows traffic to be fed according to the desires of the receiving site. If a site doesn't want binary traffic, they can have their newsfeed peers exclude the known binary hierarchies from the feed, while still receiving the text groups.

Since there are about thirty thousand newsgroups, this system would quickly break down if administrators needed to keep track of each individual binary newsgroup. So, binary groups are always placed into an appropriately-named second-level subhierarchy, for example alt.binaries. In order to feed a site only the non-binary Usenet traffic, a news administrator simply excludes those known hierarchies from a feed.

But alt.chello.binaries has “binaries” in the name, what's the problem?

It's not enough for a newsgroup to simply have “binaries” in its name somewhere. Administration of the Usenet namespace is done by hierarchy. You generally include or exclude groups based on where they fall in the namespace, not with arbitrary components in their names.

Using arbitrary name components would break down almost as quickly as needing to keep track of individual groups. Binary hierarchies are placed at the second level of the namespace (or, in theory, the top level, though this has not been done) in order to allow administrators to easily choose what they want to carry, or what they are able to carry.

In addition, a news server's spool is managed by hierarchy as well. The binary groups take up far more disk space than the text groups, so in order to keep things under control, a news administrator allocates storage for binary groups differently than for text groups. A server receiving a full newsfeed, which has the bandwidth available to take it all, still must allocate storage in this manner, and this is typically done by hierarchy.

If you post a binary to a non-binary group specifically because the binary groups expire too soon, you are missing the point. By doing this, you are taking up space which was meant for text groups. Since the text groups are very small, and don't require that much storage space, this can quickly result in disasterous consequences for the text groups - either drastically reduced retention, or a full spool (which causes the server to stop taking feeds completely).

You can't tell me what I can and can't post!

Usenet is, and always has been, a cooperative network. It continues to work only because the sites which are part of the network cooperate with each other to make it work. Without that cooperation, the network ceases to function.

When someone abuses the common resource in such a way that it damages the ability of others to use it, the other sites on the network must respond by stopping the abuse. There is no central, controlling system on Usenet to make sure things keep working as they should. Each site on the network shares in the responsibility of keeping the network running.

Misplaced binary posts are a threat to the operation of the network. Therefore, a site which allows people to post them is not fulfilling its responsibility to the network.

alt.* is an anarchy, you can't make rules for it!

The alt.* hierarchy was never intended to be an anarchy. It was created in 1987 as an alternative to the group creation process of the Big-8. This means it's easy to create a new group there. It doesn't mean that none of the rules apply.

It has never been the case that alt.* is a total anarchy. That's a myth spread by people who don't know what they're talking about.

If the group's charter says binaries are allowed, they're allowed.

Newsgroup charters don't get to allow binaries. They can't override the rules of the network.

In alt.* groups in particular, charters can be written by anyone, and are not subject to any kind of review or approval, so they have very little meaning at all. But even in more organized hierarchies, a charter is still subject to the overall rules of the network. If someone wants to create a newsgroup with very different rules, they can create it on a private server, but not as part of Usenet.

Anything a charter has to say about binary posts is meaningless. Whether binary posts are allowed is determined by whether the group is under a second-level binary hierarchy, and nothing else. Since Usenet is a cooperative network, this requirement is necessary to keep it functioning; if a group charter could overrule anything its writer wants, then the network would in practice have no rules at all and could not function.

I pay for Usenet, I can post what I want.

This is simply not the case. You may pay your provider for access to their servers, but you do not pay for Usenet. When you post to Usenet, your article is sent around the world to all the other servers on the network, using their bandwidth, and their storage space. Those sites are allowing you to use their resources, for free, in exchange for the same consideration from other sites - it's all part of the cooperation that keeps things working.

Many sites do not want binary traffic, often because they cannot afford to carry it. By posting a binary in the wrong place, you are abusing the system by forcing traffic on them that they don't want. Binary posts are far larger than text posts, so this can amount to a huge volume of data, enough to force those sites off the network completely. And you aren't paying any of them.

The people who do pay for the network, the people who keep it running, who allow you to access this grand network we call Usenet for free, don't much care what you want to post, they ask only that you post it in the right place so they can keep Usenet working. Is that really so much to ask? The alt.binaries hierarchy is open; you can create any newsgroup you want there. Is it such a burden to ask that, if you're going to be posting binary files, that you keep them where they belong?

Sometimes binaries are useful as part of a discussion.

Of course this is true. However, there is still no good reason why the files must be posted in the text groups. At a minimum, you can post a binary to an appropriate binary group, and post a pointer in the text group saying what you've posted and where you've posted it.

You can also post a file on a website and then simply post the URL in the newsgroup. This makes a whole lot more sense, because then it can truly only be downloaded by those people who want it. Files can be downloaded from websites in raw binary form, which is faster and uses less bandwidth than downloading an encoded file from a newsgroup. Free webspace is readily available, so it doesn't even have to cost you anything.

If the topic you want to discuss really does benefit from frequent binary posts, then simply create the newsgroup in a binary hierarchy. There is no reason a discussion group cannot exist in a binary hierarchy. Such a group could contain text discussion threads and binary posts.

Many groups in alt.fan.* really are binary groups.

No, they're not. Perhaps people post binaries in them, but those people are abusing the network just as much (possibly worse) than spammers.

If you want a group to post pictures of your favorite celebrity, it can be created under alt.binaries just as easily as alt.fan.

We moved to a stealth group to escape the spam.

The binary erotica groups are often flooded beyond usefulness with sex spam. However, moving to a group without sex-related keywords in the name works equally well when the new group is also a binary group.

Someone may have told you that you can't do what you want because you would be creating a so-called “stealth erotica” group. However, there has never been any kind of consensus that newsgroups for adult material must be placed in certain hierarchies, so just ignore those people.

What about alt.sex.pictures?

The alt.sex.pictures newsgroups were created many years ago, before the binary traffic became anything like what it is today. Those groups predate the current hierarchy convention for binary groups, and are now defunct. Once the binary traffic increased to the level where it needed to be dealt with in some way, alt.binaries was created.

These groups should be considered to no longer exist.

But there are things in your so-called “misplaced” groups that I want.

Tell the posters to post them in the right place, then. They are abusing Usenet, as badly as any spammer, so the fact that you want the posts doesn't make it okay. It's not about who wants the posts, it's about keeping the network running.

No one is trying to eliminate the posts, or tell you that you can't post them, or tell you that you can't have them. Only that they should be posted properly.

Providers are just cutting content to save on bandwidth costs!

This is not the case. There is no savings to the provider in moving binary posts into the proper hierarchy; the posts will still be there. Insisting that binaries be posted in the right place is very different from saying they can't be posted at all.

There is no “grand conspiracy” here to censor unwanted content. The “censors” are the people posting the misplaced binaries, and the people running the servers allowing them to be posted, preventing thousands of people from using text groups in the way they were intended.