text:
[email] (Joe Greco) wrote:

Yeah, it is really cool how that works since you can use any server
which answers ARTICLE <x@y> commands (and just about every news server
does that).  I remember Matt's postings when he was first talking about
this and how excited he was about how it was all coming together.

So the trick to making it work nicely is to adjust the way the overview
info is expired so that it goes away before the last copy of the article
vanishes.  Hmmm.  That bothers me, but the decoupling you get by "just
not bothering" is really cool too.  I'll have to think more about that.

How do the readers know which of say 10 boxes to get articles from?  Does
each reader have a priority list of what to try first?  Can you tell
the reader something like, "for alt.* try these spools in this order, but
for anything else, try these?".

So it sounds like your(/Matt's) design is based on very loose coupling
between the systems.  And you have to custom tune each box on it's own
to some extent.

My design leans more towards the idea that you are building "one" news
server which happens to be distributed over many machines - but they are
clearly all configured and managed as if it were "one" system, and not
a loosely coupled set of servers.  So, the idea of using someone elses
server as a backup just isn't part of my current design.

The separate administrative realms I think kills it, but the way I do
the spools allows them to stay very nicely in sync with the group data.
But it really limits what you can do in the sense that my design works and
acts like "one" news server, where as your design is really just a
collection of indepent servers - and servers that can work fine on their
own even while supporting the rest of the distributed system.  For example,
you could set up a small private inn server (say to run some set of local
groups).  People could read news from that server as well.  But if it also
feed it's articles to the master server, and all the readers knew to go
there to get those local.* groups), then the "big distributed" system would
look and act like it had a feed of those groups when in fact it just used
the master server. Yeah, I see there are lots of cool things you can do
with that design.

And each of your front-end readers can have it's own "view" of the world,
just like a view onto a relational DB.

So they really act just like a set of independent servers that are
sharing resources - and can, if configured that way, just act like
one big server.

Yes, that's what I did.  I added the new level of article numbering
that's done by the spools, and it does greatly reduce the flexibility
by comparison to what your system is doing, but ...

It doesn't stike me as "great quantities of complexity".  It only adds one
new concept (the name assigned to the article by the spool), and a couple
of new NNTP-like commands for dealing with the spools. (AGET and APUT
instead of ARTICLE/TAKETHIS).

But more important, it makes it work correctly.  Your system (as
found in the wild) well very likely return active file low water
marks, article counts, and xover data that do not correctly match
what's available in your spools.

My server on the other hand always returns an accurate low water
mark, and accurate xover data, accurate "listgroup" data, etc.  And the
article count is also _exactly_ correct for all but the largest groups
(like my control.cancel which currently has over 1.6 million articles in
the group).

It's really no problem.  Your server is the one that can't deal with
complex sets of articles, mine works fine.  I'll explain.

Right now my server puts each article into only one spool.  So there's
no redundancy.  But the plan is to create mirrors of the spools.

The articles are given a "name" based on what spool it was placed
in (i.e. spool 10), and what number the spool assigned that article,
say article 445566.  It's this name (10-445566) that defines the
article in all the indexes and the history file.  That name is all you
need to retrive the article (like your Message-ID).

The advantage of that name is that it becomes real simple to tell
if the article has been expired yet or if it's still available.
Spool 10 "tells" everyone what it's current range of articles
is every time it changes. (e.g., spool 10's is currently holding
articles 124456 through 999435).  So to see if article 10-445566
has been expired, you just get the current low and high article
numbers for spool 10 and see if it's within the range.  The master
server (which is acting as the reader as well right now) just
always "knows" what the ranges are for all the spools - so to
check an article to see if it's valid, you just make a quick scan
thought that small table.

When I add mirrors - they can be of different lengths.  There could
be mulitple short term mirros which are used to help distribute
the load, and one long retention spool mirror for spool 10.  As long as
article 10-445566 is still in at least one spool, then it's still
available.  And my system can tell that very quicly and very acurately
because of the way the articles get numbered sequentialy as they
are stored in a spool.

And, for example, I can do things like send all articles which are
cross posted to more than 4 groups to a special short term "spam"
spool. They will expire from the groups much faster than all the
other news - leaving holes in the groups. But even with all these
holes, my server can still acurattly tell you how many articles
are "really" in the group, and they correctly return xover data
leaving those articles out of the list.

But your server could never deal with that.  It has no way of knowing
which of the articles expired too soon.  So it would end up returning
all this xover data for articles which no longer exist on the server.

It's not hypothetical, it's in production and it works fine.

There's two types of "overview data".  There's the index which maps
group/article_num into spool-spool_article_num, and there's the
xover data.

The indexes are easy to scan and easy to expire.  You just need that
table of information which shows the low and high article numbers
for each spool.

The xover data is another story.  I generate that on demand and cache
it (in the spools as "fake" articles).  I use the group indexes (along with
the spool range tables) to figure out what's valid and what's not.  That
way, the xover files don't have to accurately reflect the articles.  The
cached sets of articles expire at some point based on how you configured
your system (i.e. which spools did you tell the system to put the xover
cahce articles into).

They won't exctly match the articles they are caching info for, but since
they are used as the "definition" of what's there and what's not, that's
not important.

I Can't ever deal with that.  That's just something that's cool about the
way you are doing it.

Right, that would never work.  I had to come up with this new scheme
to make it practical for the master process to figure out what had expired
and what didn't.

No, it's really just as simple.  I too just lay the articles down into
files (in NNTP wire format BTW with CR LF dot CR LF between them so
you can scan the raw article files and acurattly extract all the articles).
I assign sequential numbers to them and then build a simple index that
tells you where (in byte terms) article X begins.  Makes it very fast and
simpe to extrat articles.  The spool servers don't known anything about
"groups" or message-IDs. (though it would be real simple for each of my
spool servers to also maintain a history file (a.k.a. message-id index) for
it's articles so it could respond to an "ARTICLE <x.y>" request...)

Yeah, my spools are really tied to my front end boxes.  Matt made
it work with standard NNTP commands and that's just cool.

right.

Right.  It makes it real simple to figure out which articles are still
in the spool, but takes away all those fun things you can do.

And it limits you in the type of games you can play with retention of
different types of articles. And somewhat important, if you try to use
a server which is doing a tricky expire game (like my >5 x post exmple),
then there's just no way for you to deal with it.  Users will end up
getting lots of "article not there" messages.  But as long as you have
at least one long retention server that's not playing any of those
games, then it makes little differenc what the shorter term servers
do -- so that's cool and sounds like it can be very workable (though it
does make the job of the admin more complex - they really have to
understand all this complexity to get it set up right).

Yeah, me too.

If the definition of "when it's availble" stays on the master then
there's no window.  If it's distributed, then you are going to get windows.
Having N distributed processors get together and "agree" on something (like
_now_ is the time to announce the existance of this article), is one of the
clasic problems of distributed computing.  I think the answer is that it's
imposible to do without creating those windows.

I haven't thought too much yet about which route I would go -- though I was
leaning towards letting the one master maintain the group indexes and each
reader box would have to check back with it for current group status
info.  That would keep everything nicely in sync and really make it look
like one large consistant server - but puts that more resonspibilty (and
work) on the master.

Oh, my God...  Didn't even think about that problem for you.

The only way for an NNTP newsreader to map a Message-ID back to the
group/article number is to use the XRef: header. (like when
trying to follow a thread back across newsgroup bounds).  If you
just punt and leave it out then that's something else that's
broken about your server (at least in my eyes).

No, I take that back, you can use the Newsgroup: line, and then
down load all headers from the newsgroup and go searching for
a match.

So the question becomes, how many current newsreaders need that, and
what do they do if they don't get it?  I have no idea what the answer
to that is, but I suspect most newsreaders will deal with because
the Xref header really is optional - and many newsreaders my not even
try to use it (i.e. they eithre provide no easy way to follow a
thread into the old group, or they know how to use the newsgroups:
line to do it when the Xref: header is missing..).

Then don't take out the back-end master.  The spools do header
feeds to a scond master which removes dups as before (it just doesn't
xref any more), and then feed the readers as before.

Or for more redundancy at the cost of a bit more traffic, have two
back-end masters and every spool feed both of those (with IHAVE/TAKETHIS
so the redundancy just ads an extra IHAVE for each article header, not
another entire copy). and then every reader gets two feeds (but only two),
one from each of the redundant back-end masters.

Yeah, it really is cool.  The downsides that I see are the fact that
you can't deal nicely with complex expire logic on the spools (or at
least you must have at least one long-term spool with simple expire logic
that's easy to "match" with the expire logic of the readers), and this xref
issue which seems possible to get right if it's important enough to
justify.

Cool stuff.


author: bill davidsen

subject: Re: Musings upon the concept of creating reliable news services with redundant sets of unreliable servers....

text:
In article <378d4353$0$[email]>,
Joe Greco <[email]> wrote:

| Yeah, I considered that for some other purposes...  using a Promise (I
| think) IDE controller in addition to two on-boards, one can get four
| IDE channels and eight IDE drives on a single system, which reduces the
| cost overhead of the PC somewhat.  I still like SCSI better, since I can
| put hundreds of drives on a single machine, and I have a lot more faith
| in SCSI to handle server-class loads.

Agreed. The problem with IDE is that it doesn't have transfer and
disconnect, so once you issue a read on one drive the other drive on
that string is blocked until the read completes. The write situation is
far less grim, since the cache on most IDE drives is adequate to hold a
write, and it "completes" at bus speed most of the time.

I like the Promise cards, I got one on spec and plugged it in, prepared
to perform heroics to get Linux to use it. But it came up by itself,
used a single IRQ for both channels, and Linux knew all about it. I'm
about to try an ABIT PB6, which has four channels on board, hopefully
all usable.


author: Kjetil Torgrim Homme

text:
[Brad Knowles]

I was wondering about that, too.  If the newsreader uses the
Xref-value it got from overview, things are hunky-dory since the
header feed is consistent.  If it uses the value from the article, the
wrong articles will be marked read.  In order to implement quick
"catch up" functionality, it is likely that clients _will_ use
overview, though.

You give it a list of servers to try in dserver.hosts:

  nntp1	s1
  nntp2	s1
  nntp3	s2
  nntp4	s2
  master s3 o1

This will pick either "nntp1" or "nntp2" randomly (both having
priority 1 for fetching articles), and if both are down, try "nntp3"
or "nntp4", or even "master".  POST will always go to "master".

If you use different priorities, you will want different ordering on
each reader host, of course.

Diablo will ask the other servers if it has "expired" at the first
server.  You don't need to keep the same set of articles on each
storage server.

Diablo is very cool.  If it supported "never" expiry on our local
groups, our INN server would have been replaced a long time ago :-)

Kjetil T.


author: Kai Henningsen

subject: Re: Musings upon the concept of creating reliable news services with r

text:
[email] (Brad Knowles)  wrote on 15.07.99 in <[email]>:

Actually, I believe the dogcow had it's first appearance as a hidden
resource in Technote 31 in some TN Hypercard stack, or something like
that. It's a lot older than Claris. The name "Clarus", though, is, IMHO,
recent.

Kai


author: Joe Greco

text:
In news.software.nntp article <19990715020107.911$[email]>, [email] (Curt Welch) wrote:
:[email] (Joe Greco) wrote:
:
:> :How do they know
:> :when an article has expired from a spool server?
:>
:> They don't.  It is actually an extremely elegant design, because it
:> allows so much flexibility.
:
:Yeah, it is really cool how that works since you can use any server
:which answers ARTICLE <x@y> commands (and just about every news server
:does that).  I remember Matt's postings when he was first talking about
:this and how excited he was about how it was all coming together.
:
:So the trick to making it work nicely is to adjust the way the overview
:info is expired so that it goes away before the last copy of the article
:vanishes.  Hmmm.  That bothers me, but the decoupling you get by "just
:not bothering" is really cool too.  I'll have to think more about that.
:
:How do the readers know which of say 10 boxes to get articles from?  Does
:each reader have a priority list of what to try first?  Can you tell
:the reader something like, "for alt.* try these spools in this order, but
:for anything else, try these?".

No, not right now.  That's another change I'd like to investigate, right
now you simply specify a preference list - but obviously the reader may
have some hints as to what group the article is in, and you can obviously
feed a group or hierarchy to a particular spool server.  Right now, it
_will_ ask all the spool servers for the article until it finds it.  The
Diablo history lookup is very cheap, though, so this is simply a bit of
inefficiency - I do not think it will actually prove to be a bottleneck
in production.  But then again, I plan to basically have a standard art
server and a bins art server, and that's as far as I plan to distinguish
things...  any additional servers will be backups.

:So it sounds like your(/Matt's) design is based on very loose coupling
:between the systems.  And you have to custom tune each box on it's own
:to some extent.

To some extent.  (Well, my readers are identical, so I have to tune _one_
box).

:My design leans more towards the idea that you are building "one" news
:server which happens to be distributed over many machines - but they are
:clearly all configured and managed as if it were "one" system, and not
:a loosely coupled set of servers.  So, the idea of using someone elses
:server as a backup just isn't part of my current design.

That much is readily apparent.  :-)

:> Realistically, there is no good general way to keep the overview and
:> spools completely sync'ed and yet maintain the ability and flexibility to
:> keep them on separate machines or even separate administrative realms.
:
:The separate administrative realms I think kills it, but the way I do
:the spools allows them to stay very nicely in sync with the group data.
:But it really limits what you can do in the sense that my design works and
:acts like "one" news server, where as your design is really just a
:collection of indepent servers - and servers that can work fine on their
:own even while supporting the rest of the distributed system.  For example,
:you could set up a small private inn server (say to run some set of local
:groups).  People could read news from that server as well.  But if it also
:feed it's articles to the master server, and all the readers knew to go
:there to get those local.* groups), then the "big distributed" system would
:look and act like it had a feed of those groups when in fact it just used
:the master server. Yeah, I see there are lots of cool things you can do
:with that design.

Yup!  Which is why I don't mind working around the limits imposed by the
strict separation that comes with this.  It is a truly clever yet simple
way to do it.

:And each of your front-end readers can have it's own "view" of the world,
:just like a view onto a relational DB.
:
:So they really act just like a set of independent servers that are
:sharing resources - and can, if configured that way, just act like
:one big server.

Or, a little of both...  something else I am working on.

:> You can design more complex mechanisms that greatly reduce your
:> flexibility in what clever things you can do...
:
:Yes, that's what I did.  I added the new level of article numbering
:that's done by the spools, and it does greatly reduce the flexibility
:by comparison to what your system is doing, but ...
:
:> but this adds great
:> quantities of complexity.
:
:It doesn't stike me as "great quantities of complexity".  It only adds one
:new concept (the name assigned to the article by the spool), and a couple
:of new NNTP-like commands for dealing with the spools. (AGET and APUT
:instead of ARTICLE/TAKETHIS).
:
:But more important, it makes it work correctly.  Your system (as
:found in the wild) well very likely return active file low water
:marks, article counts, and xover data that do not correctly match
:what's available in your spools.

I don't know that that is actually true.  I think that the documentation
is a bit weak right now, but properly documented, I don't think it would
be a major issue.  Even if it is, 9 of 10 people are reading the most
current stuff, not the older stuff.

I'll also note that it would be possible to implement something in the
reader code to do something akin to the spool directory scan in inn1.*,
using a binary search on the overview to find the extent of existing
articles on the lower end.  This would possibly be okay as long as the
number of spool servers wasn't too large.  Maybe it could be some sort
of optional behaviour.

:My server on the other hand always returns an accurate low water
:mark, and accurate xover data, accurate "listgroup" data, etc.  And the
:article count is also _exactly_ correct for all but the largest groups
:(like my control.cancel which currently has over 1.6 million articles in
:the group).
:
:> What if you have lots of "small" spools,
:> each of which hold the last week of news, but you choose to have one
:> "retention" spool in your company that holds the last month of news?
:
:It's really no problem.  Your server is the one that can't deal with
:complex sets of articles, mine works fine.  I'll explain.
:
:Right now my server puts each article into only one spool.  So there's
:no redundancy.  But the plan is to create mirrors of the spools.
:
:The articles are given a "name" based on what spool it was placed
:in (i.e. spool 10), and what number the spool assigned that article,
:say article 445566.  It's this name (10-445566) that defines the
:article in all the indexes and the history file.  That name is all you
:need to retrive the article (like your Message-ID).
:
:The advantage of that name is that it becomes real simple to tell
:if the article has been expired yet or if it's still available.
:Spool 10 "tells" everyone what it's current range of articles
:is every time it changes. (e.g., spool 10's is currently holding
:articles 124456 through 999435).  So to see if article 10-445566
:has been expired, you just get the current low and high article
:numbers for spool 10 and see if it's within the range.  The master
:server (which is acting as the reader as well right now) just
:always "knows" what the ranges are for all the spools - so to
:check an article to see if it's valid, you just make a quick scan
:thought that small table.
:
:When I add mirrors - they can be of different lengths.  There could
:be mulitple short term mirros which are used to help distribute
:the load, and one long retention spool mirror for spool 10.  As long as
:article 10-445566 is still in at least one spool, then it's still
:available.  And my system can tell that very quicly and very acurately
:because of the way the articles get numbered sequentialy as they
:are stored in a spool.
:
:And, for example, I can do things like send all articles which are
:cross posted to more than 4 groups to a special short term "spam"
:spool. They will expire from the groups much faster than all the
:other news - leaving holes in the groups. But even with all these
:holes, my server can still acurattly tell you how many articles
:are "really" in the group, and they correctly return xover data
:leaving those articles out of the list.
:
:But your server could never deal with that.  It has no way of knowing
:which of the articles expired too soon.  So it would end up returning
:all this xover data for articles which no longer exist on the server.

I guess that doesn't bother me since I have no interest in making spam
articles available, and filter them at the head.

I can make the same case for Diablo, however, because it is possible to
move articles around at a later date - something it sounds like your
setup would have problems with (think of aging buckets).

If it really bothers somebody (like, maybe, me, since I thought about it
and then decided it wasn't really an issue in my environment), in a
single administrative realm it certainly _is_ possible to know what
articles are just about to be expired - modify dexpire to tell you, easy -
and one could extend dreaderd to have a fast "EXPIRE group:num" command
available via NNTP.  This still leaves the _possibility_ of windows, but
if I studied your setup I could probably find some odd cases there too.

This is, certainly, a bit more heavyweight than your solution, but not
horribly so.

:> Even assuming you had some hypothetical package that allowed the
:> overview database to know when the articles had been removed from
:> the spool,
:
:It's not hypothetical, it's in production and it works fine.

It _is_ hypothetical in the average user's case.  Even INN didn't do
this; maybe things are different in INN2.

:> the overview database would still have to be clever enough
:> to wait for the last instance of the article to vanish from the spool
:> servers before removing the overview data.
:
:There's two types of "overview data".  There's the index which maps
:group/article_num into spool-spool_article_num, and there's the
:xover data.
:
:The indexes are easy to scan and easy to expire.  You just need that
:table of information which shows the low and high article numbers
:for each spool.
:
:The xover data is another story.  I generate that on demand and cache
:it (in the spools as "fake" articles).  I use the group indexes (along with
:the spool range tables) to figure out what's valid and what's not.  That
:way, the xover files don't have to accurately reflect the articles.  The
:cached sets of articles expire at some point based on how you configured
:your system (i.e. which spools did you tell the system to put the xover
:cahce articles into).
:
:They won't exctly match the articles they are caching info for, but since
:they are used as the "definition" of what's there and what's not, that's
:not important.
:
:> What about a heterogeneous
:> environment where one of the spool servers isn't even the same software
:> package?
:
:I Can't ever deal with that.  That's just something that's cool about the
:way you are doing it.
:
:> :Is there some regular overview expire process which scans everything on
:> :a regular basis?  Or do the readers just expire their data based on
:> :their own logic (which might or might not match the back-end servers),
:> :or what?
:>
:> Based on their own logic... which is theoretically based on what the
:> admin knows about the actual environment.  Querying each of potentially
:> dozens of spool servers to make sure that an article no longer exists
:> on any of them would be bad for a single reader, and disastrous for a
:> large number of readers.  You could do it, sure, but I don't think you
:> would _want_ to.
:
:Right, that would never work.  I had to come up with this new scheme
:to make it practical for the master process to figure out what had expired
:and what didn't.

Something more along the lines of what I described does the same basic
thing with less gyrations up front.  I'm not saying your scheme is bad, but
it _is_ rather propietary, therefore it is something you are going to end
up supporting yourself, which is a scenario I didn't want to find myself in.

I also didn't want to need any monolithic "master process" ...   um... hey.

Now that's an idea.

I don't really like the idea of doing what I described above on general
principles because you increase complexity rapidly.  But, I wonder, if
you were just to do it on the master server, and then have the clients
use dsyncgroups to adjust the lower bounds, you can keep the low number
in sync fairly easily...  hmm.......

:> :What I did was to have each spool server number the articles
:> :sequentially. And then, instead of requesting articles by message-id,
:> :you get them by their number.  The storage scheem the servers use are
:> :set up to make this simple and fast for them to do.
:>
:> Diablo makes it fast and easy for them to request by MID.  Different
:> way to implement the same general thing.  However, if you mean that you
:> are storing them by their group.name:number, then you are pushing extra
:> work up to your spool server because it is having to maintain those
:> indexes in some fashion or another.  The Diablo spool box just lays the
:> articles down and writes history entries, and retrieval is equally
:> simple because the history lookup on the MID gives you the reference
:> to the actual article, which is then trivial to fetch and send to the
:> client.  It sounds like you have to go through some more complex
:> data to get to the actual message, although if it was coded very
:> efficiently, I guess it wouldn't be an issue.
:
:No, it's really just as simple.  I too just lay the articles down into
:files (in NNTP wire format BTW with CR LF dot CR LF between them so
:you can scan the raw article files and acurattly extract all the articles).
:I assign sequential numbers to them and then build a simple index that
:tells you where (in byte terms) article X begins.  Makes it very fast and
:simpe to extrat articles.  The spool servers don't known anything about
:"groups" or message-IDs. (though it would be real simple for each of my
:spool servers to also maintain a history file (a.k.a. message-id index) for
:it's articles so it could respond to an "ARTICLE <x.y>" request...)
:
:> That sounds like more dependence than I'd personally like to see.
:
:Yeah, my spools are really tied to my front end boxes.  Matt made
:it work with standard NNTP commands and that's just cool.
:
:> Oh, wait, your spool/spool-article# is not the same as group/article#?
:
:right.
:
:> Ich.  I see what you're trying to do, but I don't think I like it too
:> much.  It lacks the simple elegance of Diablo, although it may allow
:> you a particular form of flexibility in how you choose your retention
:> policy.
:
:Right.  It makes it real simple to figure out which articles are still
:in the spool, but takes away all those fun things you can do.
:
:> I just make sure I keep the data on the spool servers longer than the
:> readers store the overview data.  I'll concede that that is tricky and
:> a bit speculative.
:
:And it limits you in the type of games you can play with retention of
:different types of articles. And somewhat important, if you try to use
:a server which is doing a tricky expire game (like my >5 x post exmple),
:then there's just no way for you to deal with it.  Users will end up
:getting lots of "article not there" messages.  But as long as you have
:at least one long retention server that's not playing any of those
:games, then it makes little differenc what the shorter term servers
:do -- so that's cool and sounds like it can be very workable (though it
:does make the job of the admin more complex - they really have to
:understand all this complexity to get it set up right).

Solvable, if you want to take a performance hit and write some code.  I
don't think it is a problem worth my time - at least not right now, but
I might consider it in the future.

:> ... I still like SCSI better ...
:
:Yeah, me too.
:
:> ... when the master hands
:> the reader a header, it becomes "available for reading".  There is a
:> potential window where it is available on A but not B, but that will
:> happen to yours too as you scale it, I believe.
:
:If the definition of "when it's availble" stays on the master then
:there's no window.  If it's distributed, then you are going to get windows.
:Having N distributed processors get together and "agree" on something (like
:_now_ is the time to announce the existance of this article), is one of the
:clasic problems of distributed computing.  I think the answer is that it's
:imposible to do without creating those windows.
:
:I haven't thought too much yet about which route I would go -- though I was
:leaning towards letting the one master maintain the group indexes and each
:reader box would have to check back with it for current group status
:info.  That would keep everything nicely in sync and really make it look
:like one large consistant server - but puts that more resonspibilty (and
:work) on the master.

That introduces a big SPOF, which is what this whole thread was about.  I
can nuke arbitrary portions of my newsreading architecture and the
remainder will continue to work as best it can.

:> :What do you do about the Xref: line?
:
:> This is one odd thing.
:
:> When you ask for "ARTICLE <x@y>",
:
:Oh, my God...  Didn't even think about that problem for you.

I didn't, either, until I actually tried it and it spat back an article
with a Path: header that didn't mention the reader's path element and
I went "hunh?"

:The only way for an NNTP newsreader to map a Message-ID back to the
:group/article number is to use the XRef: header. (like when
:trying to follow a thread back across newsgroup bounds).  If you
:just punt and leave it out then that's something else that's
:broken about your server (at least in my eyes).
:
:No, I take that back, you can use the Newsgroup: line, and then
:down load all headers from the newsgroup and go searching for
:a match.
:
:So the question becomes, how many current newsreaders need that, and
:what do they do if they don't get it?  I have no idea what the answer
:to that is, but I suspect most newsreaders will deal with because
:the Xref header really is optional - and many newsreaders my not even
:try to use it (i.e. they eithre provide no easy way to follow a
:thread into the old group, or they know how to use the newsgroups:
:line to do it when the Xref: header is missing..).
:
:> In my particular design, then, no, you don't get an Xref: line.  If that
:> bothers you, you can design it such that the spool servers are fed by the
:> master and do get Xref: lines, but then you lose the automatic
:> availability guarantee that we just discussed, and you have to make sure
:> that your spool servers will have the article.
:
:> You could do this by making the master Xref: the articles and feed the
:> spools, then have the spools provide the header-only feeds to each of the
:> readers.  But, then in my case, you are hauling more data than necessary
:> around across your network - which is why I decided against doing that.
:
:Then don't take out the back-end master.  The spools do header
:feeds to a scond master which removes dups as before (it just doesn't
:xref any more), and then feed the readers as before.

No, I mean more shuffling up front.  Right now, my geographically diverse
spools are attached directly to my geographically diverse feeders.  The
local feeder is responsible for getting the articles into the spools, and
the distant spools feed their brothers on a time-delayed basis as backup.
This means that I am only sending one full feed from each spool/transit
cluster back to the master, and the master is only sending out header
feeds.

Doing it this other way means that I have to pull all the articles to the
master - a full feed - from each transit server.  Then I have to push the
full feeds back out to the spool servers.  So, somewhere in there is an
extra full feed...

Feeding dups to dreader processes isn't a problem, by the way...  just
somewhat inefficient.

:Or for more redundancy at the cost of a bit more traffic, have two
:back-end masters and every spool feed both of those (with IHAVE/TAKETHIS
:so the redundancy just ads an extra IHAVE for each article header, not
:another entire copy). and then every reader gets two feeds (but only two),
:one from each of the redundant back-end masters.
:
:> Either way, I like the "lego effect" Matt has built into the overall
:> design of the system, you can do lots of neat and cool things.
:
:Yeah, it really is cool.  The downsides that I see are the fact that
:you can't deal nicely with complex expire logic on the spools (or at
:least you must have at least one long-term spool with simple expire logic
:that's easy to "match" with the expire logic of the readers), and this xref
:issue which seems possible to get right if it's important enough to
:justify.

... JG


text:
[Joe Greco]

With cancels and supersedes thrown in, you may get false hits.  It
would be nice if the reader could update its overview information so
that such holes could be identified locally.  I see two approaches:

  * Let the reader keep history information, mapping message-id to
    group:num.

  * Let the feeder help the reader by adding a special header to
    cancels/supersedes, e.g. Cancel-xref: group:num group:num

I'm not sure which I prefer.  Local history at the client seems
cleanest conceptually, since you can still use an INN box (or
whatever) as the master feeder for the reader.  The history
information would only be needed for cancel processing, so it wouldn't
be catastrophic if it was lost.  However, getting help from the feeder
is probably the most efficient solution.  It will have to do some
housekeeping when it receives cancel messages anyhow.

The other approach to a more reliable GROUP, is to make some
assumptions, like "no more than five articles cancelled in a row", but
this will lead to more server load, and it will never be foolproof.

Kjetil T.


author: Curt Welch

text:
[email] (Joe Greco) wrote:

And if your set-up can't deal (well) with servers that have complex
expire patterns (leaving holes in groups etc.) who cares?  That's the type
of thing you might take advantage of it it worked, but if you didn't have
it, it's not really a problem...

Sure, there are windows in mine.  The spool is the one deciding when to
expire articles - and it tells the master after-the-fact.  So there's
a window were the master might try to access an article that "should"
be there but isn't by the time the spool receives the request to read
the article (or reports something false to the user (like the low water
mark for a group).

But because I expire in chunks (instead of constantly like in a cycbuf
design) and because the window is normally very small (on the order of 10
msec), the probability of it beeing "seen" by an NNTP client is very small.

The probabibility of hitting a standard NNTP window (like you do an XOVER
command, and then 10 minutes later try to read one of those articles and
find it's gone - either expired or canceled) is far greater than hitting
one of these errors caused by the window in the design.

Since NNTP is full of these windows - and newsreaders deal with them
all the time, you can have small windows of confusion and not really
make matters any worse for the end-user.

And in your system, as long as you can make your overview expire before
(at least most) of your spools, then it will work fine as well.  But
for it to continue running smoothly the admin will have to check it from
time to time.

INN didn't do what?  report overview info correctly?  Sure it did.
It didn't update the overview files, but it didn't need to.  It used
the group directory to figure out what was available and what wasn't.
For group commands it scanned the dir to figure out low water marks,
high water marks, and the article count.  For XOVER commands it read
the overview file, but matched it agains what it had found in the
directory to prevent it from returning overview data for articles
which no longer existed.

You could just delete an article by hand and it still reported everything
correctly.

INN 2 broke all that however.  XOVER data from inn2 included info for
articles where had been canceled. XOVER data I think also included data for
articles which had already been overwritten (expired).  As far as I could
tell, the overview data was updated only daily when you ran the overview
expire stuff - and I think the version I was playing with would never
remove xover records for canceled articles.

And if you accepted cancels, that means that many binaries groups had 50%
of their articles canceled (as spam) - and when readers tried to read those
groups, every other article came up as "not available".  Totally
unacceptable.  If you ran good spam filters which took out (at least most
of) the articles before they hit the server, and/or didn't accept cancels,
then this wasn't such a bad problem.  So like your system, if you are
willing to do things the way the system needs them done, it can work ok.
But to me it seemed like a real step down from what we had with INN for
years.  Not to meantion the xover performance issue which was the real
killer for me.

Yeah, I can understand that. If I was doing this for some ISP, I wouldn't
want to get "stuck" there either (and I would never advise that they get
stuck there either unless they could justify the cost of a full time
developement staff forever).  But since I'm doing this for my own company,
and doing development is what I like doing, it's a whole different issue...

If your system had a way that it could keep the low water marks close to
correct by itslef, then that would be a big step in the right direction.
That would free the news admin from having to adjust expire parameters by
hand to try to keep them close to what the spools were doing.  And that
just sounds like a bitch to me (not so much setting it up the first time,
but the never ending job of making sure it hasn't drifted out of alignement
over time for some reason).

Yeah, the ideal was to have no SPOF of like that, and your system gets
you much closer to that.  I like that about your system.

But my master isn't as large as your reader boxes so it's not as large
of a SPOF as I suspect you think it is.

I don't keep XOVER or header data on the master - that's all in the spools.
I have the active file info (which is the same as your master SPOF), a
history file to prevent dups (same as your master SPOF), and a file I call
the "group" file which is the index that maps group:article# into
spool:spool_article_number for every article in every group.  This is the
only thing you have distributed to your readers that I keep on my master.
And on my current server, the "group" file is acutally smaller than the
history file.

The three together, active, group, and history, are less than 2GB of data
for a 10 million article spool.  And that's all that's on the master that
needs to be mirrored.  I don't have any easy way to backup that data while
the system is running, but it would be straight forward to add a dump
function to the server which would create snapshots of those files while
the system was live.  And then I just copy those snapshots to a backup
master on a daily basis to have something ready to go when the master turns
up DOA.  Add a transaction log which is sent to the backup master in
real-time and you have a sytem that would loose almost nothing if the
master vaporized on you.

But, now that I'm thinking about this (and using your system as a model),
all I have to do is set up a header-only-like feed to a backup master and
let it keep it's own copy of those files up to date like your readers do.
You know, that's really not hard at all.  The server already knows how to
do feeds, so adding the conecpt of a header only feed is only one more
config file option for the feed and about 1 if() statement in the feed
code.  Actually, I don't even need a header-only feed, I just need to know
the MID, the Xref data, and the spool#-article# info for each article (so
it really is a special transaction log feed - but very similar in concept
to a header only feed - only smaller and faster). And the back-up master
needs xref-slave mode which I haven't done but which will be no problem to
add (it actually uses the Xref: line internally to know where to "post" the
article already).

And of course this "backup master" is now the same thing as a reader box
like yours.  So with a bit more code (to deal with postings being sent back
to the master and a few issues I see with how I cache xover data), I have
working reader boxes like yours, and any of the reader boxes could become
the "master" in seconds if the master died.  It could even automatically
switch when the master died if you either had a way  for the full feed
going to the master to switch to the backup - or could afford to always
feed a full feed to a reader set up to be the hot-backup for the master -
but it would ignore the feed (just reject all articles) until it needed to
become the master...

Hey, if my transit box (providing the full feed to the master) decided the
master was dead, it could switch the feed to one of the readers.  All the
readers could be set up to "act" as master for any (full - not header only)
articles it was sent, and then forward  the header only feed to all the
other masters and readers.  But only articles which it had assigned group
numbers to would be sent to the other boxes in a header-only feed.  This
way, the masters and the readers all become the same.  There is no master
-- only a collection of master/reader boxes which are all clones of each
other.  You control who is "acting" as master just by who you feed the
articles to.

This would have to be a special transit server that knew the importance of
it's roll in selecting the master.  And the transit server would become a
SPOF, but hey, this SPOF is now only a transit server.  If it fails, you
just stop receiving or sending news for a little while until you replace it
(all the readers keep working fine).  And it has no state that needs to be
saved.  A backup transit box that had the same "full feed" defined but
going nowhere (so it's always had a backlog ready), could act at the
backup.

And I'm thinking maybe you can even do a hot-standby swap automatically
with that transit server.  But doing that correctly sounds very complex
(and maybe impossible.) If you ever end up with two master transit servers
feeding articles to two different master/reader boxes, then the shit really
would be hitting the fan.... (and the walls, and the floor, ...)

I see a few potentials for problems here I need to think through, but the
design is starting to look very interesting....

Yes, I see now.  Geographically diverse spools.  Sure, you don't want to
have to push an extra full feed anywhere.


author: Arthur L. Rubin

subject: Re: RFD: rec.woodworking.moderated moderated

text:
Vito Kuhn wrote:
(Moderated)
Thad wrote:

TO EVERYONE READING THIS!!!!!!!!!!!!!!!!!!!!!

If you intend to vote yes on this group, your email address will be
posted all over Usenet in a unmunged form

*LIKE* *THIS*

[email]            Abdul Ahmed IV
[email]             Arlen Bruno
[email]               Arturo Stromboli
[email]        Nathan Musco
[email]             Jordan Knight
[email]              Scott Keller
[email]                       Robert Wilkins
[email]                   James Weinheimer
[email]                 Niels Olof Bouvin
[email]               Lucia Vega
[email]         David G. Roth
[email]           Carlos Bolivar
[email]               Carl White
[email]          Chris Krolczyk
[email]              Seb Coleman
[email]              Fidel Rhodes
[email]               Dorothy Zbornak
[email]           Danny Wood
[email]                Mary Deleo
[email]              Demetrius Stamos
[email]                 Blanche Devereaux
[email]                 Dimitri Cestovani
[email]                Bryan Douglass
[email]                     Pan
[email]                 Elisabetta Stromboli
[email]                  Ekkehard Uthke
[email]               Elaine Gaston
[email]            Enrico Stromboli
[email]              Enzo Stromboli Sr.
[email]              Gabriel Estrada
[email]            Tristan Etovekaf
[email]                  Bradley Fogg
[email]            Hank Fung
[email]                 Giorgio Stromboli
[email]              Peter J Ross
[email]            George Preddy
[email]                  Greg Zucherman
[email]              Gina Stromboli
[email]             Hank Zanzibar
[email]       Nina Hernandez
[email]                Helen Willis
[email]              Hugo Stromboli
[email]             Pedro Alejandro
[email]              Jada Stromboli
[email]                      Jeb Simon
[email]                   Jon Knight
[email]                 John M Price PhD
[email]               Joan Posiadlik
[email]                     Joe Morris
[email]              Joey Stromboli
[email]             Jonis Cohen
[email]           Josie Lauer
[email]               Juan Vega
[email]            Julio Nostra
[email]            Enzo Stromboli Jr.
[email]                  Joe Kewfi
[email]                     Karl Heinz
[email]              Kurt Worth
[email]                 Gwen Chase
[email]            Brian Larchmont
[email]             Lee Fong
[email]             Edwin Clarke
[email]            Mabel King
[email]             Mario Stromboli
[email]           Markus Kuhn
[email]         Mark Logan
[email]             Hans Martens
[email]         Martha Moore
[email]           Joe McIntyre
[email]         Biff Mullins
[email]              Louis Browne
[email]           Hiro Niihara
[email]           Noberto Alcazar
[email]          Norman Byrne
[email]            Olivia Stromboli
[email]                  Pat Segovia
[email]           Sophia Petrillo
[email]                Sal Gallo
[email]  Ben Reese
[email]                Rhino
[email]              Rosa Stromboli
[email]                Juanita Ruiz
[email]               Ronald Weiss
[email]            Sally Parker
[email]         Ana Sbet
[email]            Sean Franklin
[email]         Sharif Qureshi
[email]      Sofie Jansen
[email]         Steven Green
[email]          Pierre Roche
[email]                 Thomas Cuny
[email]                       Ted Conklin
[email]       The Puppy Wizard
[email]              Tomasz Brzezinski
[email]                     Tom Perrett
[email]              Tony Stromboli
[email]              Robert-Jan van Ette
[email]        Lynn Vassi
[email]           Vladimir Chernova
[email]              Don Wahlberg
[email]             Mark Wahlberg
[email]           Marlon Brown
[email]            Zach Berry
[email]           Darius Polson
[email]                   Georges LaRouche
[email]                Zeke Williams

These dickheads voted yes on soc.war.consequences

This will be posted to alt.spam and misc.test too ! :)))))

If you vote yes on rec.woodworking.moderated, expect your address to be
posted to *HUNDREDS* of groups!!!! ;)


text:
Arthur L. Rubin wrote:

No, he didn't....

Reported to google, for what it's worth.


author: Lionel

text:
Kibo informs me that "Arthur L. Rubin" <[email]>
stated that:

I'd be most interested in hearing what they have to say about their
users impersonating others on Usenet. I suspect that you'll just get a
form-letter from their ignore-bot, that when translated from weaselish,
will tell you that no human has seen your report, & that they aren't
going to do anything about it. That's how they currently 'handle'
complaints about MMF-spam & AUP violations.


author: Linda Chasey Vincent

subject: On-line Access Services Group?

text:
I currently work in an organization that has direct access to the Internet,
World Wide Web, ftp, etc.  I'm planning to leave the company but would like
to maintain an on-line connection to the world out there.  Is there a group
that discusses the various connection services available like Compuserve,
America On-line, etc.  I'd like to continue the level of access I currently
have as a minimum if that's possible and would like to hear which services
people find satisfactory.

Thanks.
Linda Chasey Vincent


subject: Re: On-line Access Services Group?

text:
In article <2rqn26$[email]>, [email]
(Linda Chasey Vincent) writes:

America On-line seems to be the best service I know of.
RDC


author: Jonathan Grobe

subject: Re: On-line Access Services Group?

text:
In article <2rqn26$[email]>, [email] (Linda Chasey Vincent) writes:
alt.online-service discusses the standard online services, and there
are individual groups such as alt.online-service.america-online and
alt.online-service.delphi. Those two are the only ones who provide
Unsenet groups currently.
But if you are satisfied with what you have at the company and don't
want things like online encyclopedias you should look instead for a public
dialup access provider. A list of these list published in news.answers
and alt.internet.access.wanted is called PDIAL. If it has expired at your
site send e-mail with subject line: SEND PDIAL to [email]
Groups of interest are alt.internet.access.wanted and alt.internet.services.
Look at the FAQ in alt.internet.services. A major problem
is if the access is a local phone call. This depends on where you live.
You will have a choice from none to a few companies which provide local
phone call access. Otherwise you end up paying $6/hour or so for
800 number access. This contrasts with say paying $30/month for
unlimited usage if there is a local phone number.

-----------------------------------------------------------------------------
Jonathan Grobe  [email]


author: Cindy Bieghler

subject: Re: Skirvin KILLS Grubor and Boursy

text:
Stephen Boursy ([email]) wrote:
: In article <4hesds$[email]>, Dr. Grubor
: <[email]> wrote:
: >In <4henuv$[email]> [email] (Dave
: >Hayes) writes:
: >>[email] (Tim Skirvin) writes:
: >>>
: >>>	I'm not going to get into it all; there's no need.  I think most
: >>>of us understand what just happened.  All I really have to say here
: >>>is that I have just added Boursy to the Global Killfile.  I was wrong.
: >>>I really do believe that nobody should read his posts.
: >>

: 	From the censorous Tiny Timmy that's quite a complement!  Timmy
: finds me dangerous it seems.

: >>As if that would make any sort of difference, or be important 10,000
: >>years from now.
: >>--
: >Or like it is even important today.
: >The important thing is to classify this young boy
: >who has gotten ego-crazed and out of hand.
: >He pulls the "Skirvin back-stabber" syndrome.
: >
: >DrG
: >

: 	Well Dr. Grubor I still think we should call
: Timmys mom--she is supporting the boy and probably
: has no idea he's engaged in forgery.

: 				Steve


author: Michael Tokarev

subject: SPEWS: S2728 (mail15.com): moved to another ISP

text:
From S2728:

mail15
|--------------------
0, 195.2.82.22, test.mail15.com
0, 195.161.112.90, ns2.mail15.com
0, 217.107.216.22, ns1.mail15.com
0, 217.107.216.23, stable.mail15.com
1, 194.186.131.96, mail15.com / stable.mail15.com
1, 194.186.131.64 - 194.186.131.127, mail15.com
1, 194.186.131.0 - 194.186.131.255, sovam.com (mail15.com)
0, 194.186.0.0/16, sovam.com (mail15.com)
---------------------|

Currently, mail15.com is hosted by MTU.ru.  Starting from whois:

Registrant:
Zaitsev, Igor (TDRJYLSASD)
    Profsousnaya str., 84/32, k.602t
    Moscow
    RU

    Domain Name: MAIL15.COM

    Domain servers in listed order:

    IMAP.MAIL15.COM              62.118.249.46
    WWW.MAIL15.COM               62.118.249.44

This netblock (62.118.249.0/24, MTU-CLC3, ZAO MTU-Intel)
seems to be colo-block of MTU.ru, and contains quite some
stuff in addition to mail15.com.

While former www.mail15.com on sovam, 194.186.131.96,
while still have rDNS of www.mail15.com, does not respond
to http requests anymore.

JFYI.

/mjt


author: McWebber

subject: Re:  SPEWS: S2728 (mail15.com): moved to another ISP

text:
"Michael Tokarev" <[email]> wrote in message
news:bupsa6$1gap$[email]...

If you control 194.186.131.96 I suggest you fix that before asking for
removal.


author: Richard Johnson

subject: Re: SPEWS: S2728 (mail15.com): moved to another ISP

text:
In article <bupsa6$1gap$[email]>,
 Michael Tokarev <[email]> wrote:

Good catch, thanks.

The presence of the mail15.com gang will be quite unfortunate for
anyone else in that block.  Surely mtu.ru isn't that stupid?

Especially given the incomplete cleanup of these criminals, I have no
particular reason to delist 194.186.131.64/26.  Still it's good to know
that service -may- have ended there.  Thanks for noticing that.

Richard


text:
McWebber wrote:

Yup a good suggestion...  But I don't control those networks ;)

Well... We attempted to request services from mtu.ru
(new home of mail15.com) but I convinced our folks to
not to do that - exactly because of poor spam record
of mtu.ru, without any hope of cleaning stuff up in
any reasonable future.  So we now geting services from
another ISP (who uses mtu.ru newsserver btw.. so still
somehow related to mtu.ru.. bah ;)

/mjt


author: Dolphin

text:
#begin  [email] (or was it Richard Johnson.com)
 message <burd63$l8k4m$[email]> reply:
<SNIP>

They ARE that stupid. They were notified by *another ISP*, who has
finally kicked those mail15.com spammers, before they flew to MTU,
4+ months ago!
http://groups.google.com/groups?selm=bk8saa%243m8%241%40slim.sovintel.ru

Obviously, MTU has no problems with being blocked world-wide. Or they
think that it will not happen to them (the Russian "avos'").

Dolphin.


author: Cindy Tittle Moore

subject: rec.pets.dogs:  Introduction FAQ

text:
Archive-name: dogs-faq/introduction
Version: 2.2
Last-modified: 10 July 1992

This is one (of eight) of the FAQ (Frequently Asked Questions) Lists
for rec.pets.dogs.  It is posted on a monthly basis: updates,
additions, and corrections (including attributions) are always
welcome: send email to one of the addresses below.

The eight parts are all archived at pit-manager.mit.edu (18.172.1.27)
in the directory /pub/usenet/news.answers/dogs-faq.  The files are:
introduction, new, health-care, training, working, AKC-titles, misc1,
and references.  To obtain the files, first try ftp to
pit-manager.mit.edu and look under that directory.  If ftp does not
work from your site, then try the mail server: send email to
[email] with

send usenet/news.answers/dogs-faq/introduction
send usenet/news.answers/dogs-faq/new
send usenet/news.answers/dogs-faq/health-care
send usenet/news.answers/dogs-faq/training
send usenet/news.answers/dogs-faq/working
send usenet/news.answers/dogs-faq/AKC-titles
send usenet/news.answers/dogs-faq/misc1
send usenet/news.answers/dogs-faq/references

in the body of the message (leave the subject line empty).  If you
don't want all of them, include only the lines of the ones you want.
You do have to repeat the path information for each file.

Changes and additions indicated with |'s.

I.  Introduction

  A.  Introduction to rec.pets.dogs.
  B.  FAQ's for rec.pets.dogs.
  C.  Acronyms.
  D.  Abbreviations.
  E.  Glossary.
  F.  Kennel Clubs.
  G.  Mailing Lists.
  H.  Acknowledgements.

A.  Introduction to rec.pets.dogs.

Rec.pets.dogs is a newsgroup devoted to domestic canine issues.  This
group can be friendly and helpful.  Flamewars are generally limited to
several topics: crating dogs, training by the Koheler method, docking
and cropping, animal rights activism, wolf hybrids and pit-bull (or
other breed) dog bans.  New readers are advised against starting these
topics up on the newsgroup as long, protracted, and inflammatory
discussions often ensue.  The facts pertaining to these controversial
topics are covered in the FAQ's.  This is not to say that these topics
are forbidden from discussion on rec.pets.dogs, but that readers would
greatly appreciate it if you checked out what the FAQ had to say on
them to make sure you have something new to contribute.

This newsgroup was formed in the summer of 1991.  It is a splinter
group from rec.pets, which originally carried the canine as well as
other topics.  Joe DiBenedetto proposed the split and collected the
votes, which proved more than enough for its official creation.

Michael Mahler attempted a FAQ in 1991 that consisted of compiled but
unedited articles saved from the net.  Due to time constraints, he
gave up the project, and I received copies of the articles that he had
saved and incorporated much of them in here from other readers.  I
have heard of other attempts at FAQ's well, but no real details.

1.  Charter

There exists no official charter for this group.  However,
unofficially, this group exists to discuss anything related to
canines.

2.  Asking questions

By all means, ask questions on this newsgroup, that's what this is
for.  Most of us are more than happy to help you out.  There are some
things you might want to keep in mind, though:

Is it a frequently asked question?  If you have a question about
housetraining, for example, you might find the answer in one of the
group's FAQ.  It's entirely possible you have a question not covered
in the FAQ, but it helps to check first.

Are you giving us enough information?  Sometimes people ask brief
questions that don't get the responses they wanted.  This is often
because the brevity of the question leads many readers to infer other
possibilities.  In particular, questions like "I want a dog that stays
outside," "I want a dog for the kids, what's a good one," "I want a
dog for protection" will generate many followups that are concerned
with whether or not you really should get a dog, along with
recommendations of books that you should read first.  If you've
already done your homework, by all means, take the extra time to
describe that in your request so that you can get more useful
responses.

B.  FAQ's for rec.pets.dogs.

1.  Eight monthly posted FAQ's

There are eight FAQ's, split up by topic rather than by length.  All
eight are posted regularly.  The first one, "Introduction," is this
file, which gives a brief description of this group, details common
acronyms, includes a glossary, describes several national kennel
clubs, and thanks the people who made this all possible.  The second,
"New Owners, New Dogs," is aimed at new dog owners and puppies.  Thus
it contains information on the needs of puppies and helps the new or
inexperienced owner get started.  The third, "Health Care Issues," is
a good look at canine health, including diseases (prevention,
treatment and prognosis), poisons, medication, etc.  The fourth FAQ,
"Dog Behavior, Problems and Training," outlines the way dogs think and
presents common behavioral problems along with suggested constructive
solutions.  In addition, it dicusses training methods, communication
with your dog, and suggests ways for teaching basic commands.  The
fifth FAQ, "Companion and Working Dogs" covers the amazing variety of
things that dogs can do.  The sixth, "American Kennel Club Titles,"
covers AKC test requirements and discusses showing.  It also briefly
discusses breed clubs and titles obtainable through other clubs.  The
seventh FAQ, "Assorted Topics," contains a variety of topics that did
not fit any particular theme, such as wolf-hybrids, breeding, and
allergies.  The last and eighth FAQ is a simple compendium of the
references that are liberally sprinkled throughout the other FAQ's,
plus additional books, articles, catalogues, mail-order companies and
organizations.

There is certainly room for more FAQ's -- the world of dog fanciers is
a wide one indeed!  In general, I welcome suggestions on ways to
improve FAQ's or construct new ones.  I am particularly appreciative
of written-up contributions.  The format of the FAQ's has been chosen
in particular to provide individually useful files.  Each of the files
is also less than 65K (or they are as close as possible) in order to
meet mailer standards.  Most mailers can handle 100K or more, but
there are some out there that have a limit of 65K, so I tried to
accomodate those.  At this point, it looks like I may put together
a second part to the Assorted Topics FAQ unless I can think of a
way to regroup them under something more specific.

2.  Table of contents across the FAQ's

I.  Introduction

  A.  Introduction to rec.pets.dogs.
  B.  FAQ's for rec.pets.dogs.
  C.  Acronyms.
  D.  Abbreviations.
  E.  Glossary.
  F.  Kennel Clubs.
  G.  Acknowledgements.

II.  New Owners, New Dogs.

  Prologue.
  A.  What Kind of Dog?
  B.  What are My Responsibilities?
  C.  Where?
  D.  Veterinarians.
  E.  Puppies.
  F.  Puppy-Proofing Your Home.
  G.  Feeding Your Puppy.
  H.  Teething.
  I.  Chew Toys.
  J.  Preliminary Training.
  K.  Socialization of Puppies.
  L.  Housetraining Your Puppy.
  M.  Living with Other Pets.
  N.  People Food.

III.  Health Care Issues

  Prologue
  A.  In General.
  B.  Dental Care.
  C.  Trimming Nails.
  D.  Neutering.
  E.  Bathing.
  F.  Skin Problems.
  G.  Vaccinations.
  H.  Canine Ailments.
  I.  Disease Transmission (Zoonoses).
  J.  Worms.
  K.  Pills and Dosing.
  L.  Vomiting.
  M.  Poisons.
  N.  Aging.

IV.  Dog Behavior, Problems and Training.

  A.  Understanding Your Dog.
  B.  Principles Behind Dominance.
  C.  Dominance Problems.
  D.  Housetraining Problems.
  E.  Submissive Urination.
  F.  Other Common Problems.
  G.  Comments on Obedience Training
  H.  Attention.
  I.  Corrections.
  J.  Using Food.
  K.  Training and Corrective Collars.
  L.  Learning to Sit.
  M.  Down, Stay, Off.
  N.  Heeling.
  O.  Learning Recall.
  P.  Beyond Basic Obedience.

V.  Companion and Working Dogs.

  A.  Dogs for the Blind.
  B.  Hearing and Signal Dogs.
  C.  Canine Companions for Independence.
  D.  Search and Rescue Dogs.
  E.  Water Rescue Dogs.
  F.  Narcotics and Evidence Dogs.
  G.  Schutzhund.
  H.  Police Dogs.
  I.  Sled Dogs.
  J.  Gaming Dogs.

VI.  American Kennel Club Titles

  A.  Showing.
  B.  Conformation.
  C.  Obedience.
  D.  Field.
  E.  Tracking.
  F.  Herding.
  G.  Championships.
  H.  Other AKC Titles.
  I.  AKC Breed Clubs.
  J.  Other Organizations.

VII.  Assorted Topics.

  A.  A New Baby.
  B.  Breeding.
  C.  Crate Training.
  D.  Docking and Cropping.
  E.  Facts and Opinions about Neutering.
  F.  Finding a Home for a Dog.
  G.  Invisible and Electric Fences.
  H.  Owner Allergies.
  I.  Pet Doors.
  J.  Pet Identification and Theft.
  K.  Pet Insurance.
  L.  Pet Sitting and Kennel Services.
  M.  Record Keeping.
  N.  Removing Odors and Stains.
  O.  Travel and International Travel.
  P.  Wolves and Wolf Hybrids.

VIII.  References.

  A.  Books.
  B.  Articles.
  C.  Companies and Catalogues.
  D.  Magazines.
  E.  Organizations.

3.  FAQ's for particular breeds

In addition to these monthly posted FAQ's, there are available FAQ's
for specific breeds.  These are posted rarely, perhaps once or twice a
year.  They are available via anonymous ftp to pit-manager.mit.edu
(18.172.1.27) under pub/usenet/news.answers/dogs-faq/<breed>.  For the
exact file name, check the name in parentheses after the breed.  If you
do not have anonymous ftp, send email to [email]
with "send usenet/news.answers/dogs-faq/<breed>" in the subject line,
leaving the message body empty.  For example, to reference Bloodhounds,
you would use usenet/news.answers/dogs-faq/bloodhounds.

If you do not see your favorite breed below, send me email if you want
to put one together yourself!  (Addresses are at the end of this
post.)  I only wrote one of these, as a guide for the rest of them.
These FAQ's have been contributed by fellow rec.pets.dogs readers.

Currently, FAQ's are available for:

Bloodhounds (bloodhounds).
Collies (collies).
Greyhounds (greyhounds).

C.  Acronyms.

New readers frequently have trouble understanding the acronyms that
have developed in this group.  Other acronyms are widely used among
dog fanciers.

A partial listing (feel free to send in contributions):

1.  General acronyms

AHBA    American Herding Breeds Association.
AKC:    American Kennel Club.
|AWTA:   American Working Terrier Association.
ALF:    Animal Liberation Front (USA animal rights activist group).
ASCA:   Australian Shepherd Club of America
BIS:    Best in Show.
BOB:    Best of Breed (male or female).
BOS:    Best Opposite Sex (to BOB).
BW:     Best of Winners.
CEA:    Collie Eye Anomaly.
CERF:   Canine Eye Registration Foundation: registers "normal eyed dogs".
CHD/HD: Hip Dysplasia.
CKC:    Canadian Kennel Club.
CPI:    Canine Parainfluenza.
CPV:    Canine Parvovirus.
DHLPP:  Distempter, Hepatitis, Leptospirosis, Parainfluenza, Parvovirus shot
FDSB:   Field Dog Stud Book; a registry of pointing dogs published by the
        American Field Publishing Company.  Many pointing dogs are dual
        registered AKC/FDSB.
GSD:    German Shepherd Dog.
IM(H)O: In My (Honest/Humble) Opinion.
KC:     Kennel Club (of Great Britain).
KPC/T:  Kindergarten Puppy Class/Training.
NQ'd:   Not Qualified -- disqualified from the ring
OES:    Old English Sheepdog.
|OCD:    Osteochondrosis Dissecans (separation of joint cartilage)
OFA:    Orthopedic Foundation for Animals (hip certification).
PETA:   People for the Ethical Treatment of Animals (USA animal rights
        activist group).
PRA:    Progressive Retinal Atrophy.
|RBKC:   Rare Breed Kennel Club.
ROM:    Register of Merit, awarded by parent clubs of specific breeds.
RWB:    Reserve Winner's Bitch.
RWD:    Reserve Winner's Dog.
|SKC:    States Kennel Club (mainly active in the South and New England)
UKC:    United Kennel Club.
WB:     Winner's Bitch.
WD:     Winner's Dog.
WH:     Wolf Hybrids.

2.  Titles

CD:     (AKC) Companion Dog.
CDX:    (AKC) Companion Dog Excellent.
UD:     (AKC) Utility Dog.
OTCH:   (AKC) Obedience Trial Champion.

JH:     (AKC) Junior Hunter.
SH:     (AKC) Senior Hunter.
MH:     (AKC) Master Hunter.

TD:     (AKC) Tracking Dog.
TDX:    (AKC) Tracking Dog Excellent.

UDT:    (AKC) Utility Dog Tracking.

HT:	(AKC) Herding Tested.
PT: 	(AKC) Pre-Trial Tested.
HS:	(AKC) Herding Started.
HI:	(AKC) Herding Intermediate.
HX:	(AKC) Herding Excellent.
HCH: 	(AKC) Herding Champion.

FC:     (AKC) Field Champion.
AFC:    (AKC) Amateur Field Champion.
NFC:    (AKC) National Field Champion.
NAFC:   (AKC) National Amateur Field Champion.

CGC:    (AKC) Canine Good Citizen.

HIC, HC (AHBA) Herding [Instinct] Certified.

|CG:     (AWTA) Certificate of Gameness, soon to be annexed by the AKC
|WC:     (AWTA) Working Certificate (only dogs who go to ground)
|AGHC:   (AWTA) Above Ground Hunting Certificate

CH:     Champion, usually with country in front of it, e.g., Br. CH, Am.
        CH, Can. CH.  (AKC Champion of Record).

Sch I,
II, III (Schutzhund) consecutive working titles
FH:     (Schutzhund) Fahrtenhund (advanced tracking title, comparable to TDX)
AD:     (Schutzhund) Ausdauer (endurance test)
VB:     (Schutzhund) Traffic Secure.

TT:     (American Temperament Test Society) Temperament test.

STD:    (ASCA) Started Trial Dog (s/d/c suffix: sheep, ducks, cattle).
OTD:    (ASCA) Open Trial Dog (same suffixes).
ATD:    (ASCA) Advanced Trial Dog (same suffixes).

HR:     (UKC) Hunter Retriever
HRCH:   (UKC) Hunter Retriever Champion
GR HRCH:(UKC) Grand Hunter Retriever Champion

TDI:    Therapy Dogs International

D.  Abbreviations

Dobie: Dobermann Pinscher.
Golden: Golden Retriever.
Lab, esp. Yellow/Black Lab: Labrador Retriever.
Mal: Alaskan Malamute.
Newfie: Newfoundland.
Parvo: Canine Parvovirus.
Rottie: Rottweiler.
Sheltie: Shetland Sheepdog.
Sib: Siberian Husky.

E.  Glossary

alpha: Dogs recognize intricate social hierarchies.  The one "on top"
is "alpha."  Alpha can also be relative, denoting which of a pair of
dogs dominates the other.  Packs include humans where applicable, and
owners should always be alpha to their dogs.

bitch: A female dog.  Not used in a derogatory sense.

bitchy: A male dog showing female characteristics (e.g., in size
and weight).

bloom: Glossiness of coat.

brace: A pair of dogs.

brisket: Front of chest between forelegs.

brood bitch: Female used for breeding.

coupling: That part of the body between the shoulders and hips; some
dogs are short coupled and others are longer coupled (aka "standing
over more ground").

crabbing: A dog that moves toward you with its body at an angle
rather than straight on.  Also called side-wheeling or side-winding.

croup: The portion of the back directly above the hind legs, before
the tail.

dam: The female parent.

dewclaws: Rudimentary toes often removed shortly after birth.

dewlap: The skin on the throat.

dish faced:  Where the tip of the nose is higher than the stop.

dog: 1.  Any dog, male or female.  2. A male dog, especially when used
in conjunction with bitch.

doggy: A female dog showing male characteristics, especially in weight
and size.

eyeteeth: The upper canines.

flews: Pendulous upper lips.

heat: The mating period for bitches.

hocks:  The joints above the pasterns of the legs.

pad: The cushioned part of the foot.

pastern: The part of the leg below the knee in rear legs.

Schutzhund: A German-developed training regimen for protective work
used world-wide.  Only strong, determined, and aggressive dogs do well
with Schutzhund.  Both dogs and bitches have been trained in
Schutzhund.  German Shepherd Dogs are traditionally associated with
Schutzhund, but medium to large dogs of any breed may be so trained.

second thigh: The area of the hindquarters between the hock and the
stifle.

septum: The vertical line between the nostrils.

show dog: A dog bred for conformation to its breed standard and shown
in competition on that basis.

sire: The male parent.

stifle: The upper joint of the rear legs, corresponding to humans' knees.

stop: The point between a dog's eyes.

stud: Male dog used for breeding.

withers: The highest point of the shoulders, right behind the neck.

F.  Kennel Clubs.

(condensed from _Reader's Digest Illustrated Book of Dogs_., pgs 368-377)

There are organizations set up around the world for animals.  Although
the character of each differs, the basic responsiblities of a kennel
club include:

  * Recognizing breeds.  The criteria for being recognized varies a
    great deal from country to country.

  * Maintaining breed registries.  Issues certificates of registration
    to owners of dogs that qualify.

  * Compiling stud books.  Traces ancestries of registered dogs as far
    back as records have been kept for that breed.

  * Establishing breed standards.  Publishes descriptions of the ideal
    dog for each breed, to help with judging breeds in show.

  * Sponsoring dog shows and trials.  Enforces procedures to be
    followed to insure merited reception of awards.

  * Approval of judges, and judging procedures.  Sets standards of
    knowledge for a judge.

Often, national kennel clubs accept as members or affiliates canine
clubs devoted to particular breeds that sponsor their own
competitions.  Other independent clubs set their own breed standards.

Although show trials are the most common, where a dog's appearance is
the sole criterion for awards, outdoor competition is also popular:
hunting, tracking, obedience, herding, and lure-coursing are some of
the alternatives.  In these only performance counts.

1.  American Kennel Club

AKC
51 Madison Avenue
New York, NY 10010

2.  Australian National Kennel Council

ANKC
Administrative Buildings
Royal Showgrounds, Ascot Vale
3032,  Victoria

Each of the eight Australian states/territories has its own canine
club, but the controlling body is ANKC.  Established in 1958, its
objective is to propose uniform policies and procedures for all of
Australia, although member clubs are not compelled to adopt any of the
procedures.  There are only minor administrative differences, however,
and champions in one state are recognized as such in others.

Major Australian show classes are
  Baby puppies (4-6 mos)
  Minor puppies (6-9 mos)
  Puppy class (6-12 mos)
  Junior class (6-18 mos)
  Intermediate class (18mos to 3yrs)
  Australian-bred class (6mos and over, born in Australia)
  Open class (any dog, generally 1yr and over)

Ear cropping is illegal in Australia, and ANKC considers dogs with
cropped ears ineligible for registration.

The judging system is based on points, accumulated from being judged
Best of Sex; the number of points reflects the number of competitors
over which the dog won (with minimum and maximum limits).  Dogs are
judged by breed standards established by the Kennel Club (Britain),
except for breeds of Australian origin.  Highest title: Breed Champion.

ANKC groups:
  Toy
  Sporting terrier
  Gun dog
  Hound
  Working
  Utility
  Non-sporting

3.  Canadian Kennel Club

CKC
89 Skyway Avenue
Rexdale, Ontario M9W 6R4

The club was formed in 1888 when Canadians became determined to have
their own club after AKC's four year inclusion of Canada in its
beginning.  The two clubs have worked closely together since then, e.g.,
reciprocal recognition of each other's stud books and judges.  There
are similar judging systems, many common regulations, and most of the
same breed standards.

The judging system is based on points.  A dog must accumulate 10
points under at least three different judges at shows held under CKC
regulations.  Points are awarded based on the number of dogs in
competition and handed to each Best of Sex winner.  Up to five
additional points may be awarded should the dog then place in its
group or win Best in Show.

Unusually among kennel clubs, the CKC is responsible to the federal
government in registering its dogs.  Selling a pure-bred puppy without
first registering it carries penalties.  Pedigreed dogs must be
identified on registration forms by either tattoo or noseprint.

CKC groups
  Sporting
  Hound
  Working
  Terrier
  Toy
  Non-sporting
  Herding
  Miscellaneous

4.  Fe'de'ration  Cynologique Internationale

FCI
Rue Le'opold-II
14B-6530 Thuin
Belgium

The FCI is a world-wide federation of national dog clubs.  Its
objective is to bring about international standards in the breeding,
exhibiting, and judging of purebreds.

Founded in 1911, it was originally European -- claiming membership
from France, Belgium, Germany, Austria, and Holland.  Today,
membership includes representatives from 19 European countries as well
as several from Latin America, Asia, and Africa.  Another 11 countries
are affiliated (including Britain and South Africa).

FCI automatically recognizes the breeds of member countries -- hence
it recognizes over 300 breeds.  Breed standards approved by FCI are
drawn up from the country of origin, helping to keep those standards
uniform.  Member countries may refuse to register dogs showing genetic
flaws or non-conformation to standards; this helps eliminate
hereditary defects.

FCI hosts two types of shows: national and international.  The country
in which the show is held determines the judging system.
International titles: International Working Trial Champion, and
International Beauty Champion, both based on accumulating aptitude
certificates and prizes in several countries under several judges.
Difficulties are encountered by various quarantines of some of the
countries.

FCI groups:
  Shepherd Dogs
  Watch Dogs and Working Dogs
  Terriers
  Dachshunds
  Hounds for Big Game
  Hounds for Small Game
  Pointers (except British breeds)
  British Gun Dogs
  Fancy Dogs
  Greyhounds

5.  Kennel Club (Britain)

The Kennel Club
1-5 Clarges Street
London W1Y 8AB

The Kennel Club was formed in 1873.  Its objective is to bring under
central authority various canine clubs to ensure uniform standards,
rules, and procedures, as well as a comprehensive registry for all
breeds.  There are more than 1,700 member clubs in The Kennel Club
today.  The Kennel Club has reciprocal arrangements with more than 40
national clubs and is an associate member of FCI.

The KC does not recognise Schutzhund.  To gain a KC studbook number a
dog must get a first, second or third prize in the Limit or Open Class
at a Championship show or a CC or RCC.

The British title Champion (bench) is generally considered the most
difficult to obtain.  The dog must earn three Challenge Certificates
from three different judges (one out of puppy class). Some breeds,
those on the import register do not have CC's.  -- but CC's are
sharply limited, and even if Best of Sex is won, the judge may
withhold a CC.  It may take years to obtain the title.

Kennel Club groups
  Hound
  Gun dog
  Terrier
  Utility
  Working
  Toy

6.  Kennel Union of Southern Africa

KUSA
Box 2659
Cape Town 8000
South Africa

Organized in 1895 to unite all the kennel clubs in South Africa, KUSA
is composed of some 150 affiliated clubs.  Most of KUSA's approved
standards are from Britain's Kennel Club.  Deviations or omissions
from the KC generally follow FCI standards instead.  Of FCI members,
KUSA is the most closely associated club.  A total of 181 dogs are
recognized by KUSA.

KUSA groups
  Hound
  Gun dog
  Terrier
  Utility
  Working
  Toy

7.  New Zealand Kennel Club

NZKC
Private Bag
Porirua

Operates virtually identically to ANKC; breed standards and rules are
the same.  There is no quarantine between Australia and New Zealand
(unless recently imported from elsewhere).

8.  United Kennel Club [USA]

A smaller kennel club in the US, with an emphasis on working
competitions.

G.  Mailing Lists.

There are several email lists on canines and related topics.
See the training FAQ for the one on obedience, the AKC-titles
for the herding list.

A general canine list is: CANINE-L.  To subscribe, send email
to [email] with
subscribe canine-l <your name>
in the body of the message, where <your name> is your actual
name, not your login name or email address or anything else.

H.  Acknowledgements.

I would like to thank the following people who took the time to send
me their comments.  Many of them contributed substantial material for
this FAQ.  Some contributed unknowingly, as I lifted their posts
directly from the newsgroup, or obtained them from summaries kept by
other people.

Thanks to: Ed Anderson, David Austin, Chris Barnes, Loralie Bellows,
Marla Belzowski, Rachel Bereson, Mike Berger, Heike Berlind, Kathy
Buetow, Matthew W.  Blake, Amy Blankenship, Wade Blomgren, Mary
Blumenstock, Kathleen Brewer, Dag Bruck, Nancy Bulinski, Robin Bush,
Alex Campbell, Jose' J.  Capo', John Chandler, Bill Chapp, Jason Chen,
Joseph T. Chew, Bill Chiachiaro, Mike Clark, James R.  Coggins, Dale
Cook, Deborah J.  Cooper, Steve Crumley, Schalene Dagutis, Brad
Davidson, Barbara Davis, Jack Dean, Len DePalma, Len DeRoeck, Dennis
Doubleday, Kim Dyer, Damon Feldman, Lindy Fleck, Debbie Forest, Dan
Fowler, Del Fredricks, Carole Susan Fungaroli, Neal Gaborno, Michael
Garzione, Patricia Geschwent, Rob G.  Glenn, Robin Goldstone, Jim
Graham, Marsha Jo Hannah, Tom Haapanen, R.  Hardin, Terri W. Hardwick,
Sharon Hasting, John Heim, Julie Hill, R. G. Hoff, Duke P. Hong, Sue
Utter Honig, Gerald Hough, Terry Hull, Sally A. Hundley, Larry Hunter,
John Interrante, Jane Jackson, Victoria Janicki, Tor Erik Jeremiassen,
Marget Johnson, Debbie Keyes, Joyce Andrews King, Erich Klinghammer,
Rich Kulawiec, Virginia P. Kunch, Kimberly Laird, Karina Laramie,
Stephen N.  Larson, Stephen R. Lee, William LeFebvre, David E.  Levy,
Bill McCormick, Allen McIntosh, Marian McKenzie, Meg McRoberts, Ellen
M. McSorley, Michael Mahler, Dana Massey, Douglas Merrill, Steve
Michelson, Liza Miller, Carol Miller-Tutzauer, William L. Millios,
Wendy Milner, Maralee Morado, Gary Morris, Kathy Moser, S.  Mudgett,
Lily Mummert, Anne Myers, Victoria Neff, Barbara A. Neves, Devon
Nichols, David Northrop, Eugina M. Novak, Jeff Parke, Brian Pawlowski,
Jill N.  Pappenhagen, Jon Pastor, David Prager, Nancy Rabel, Margaret
F. Riley, Nancy Roberts, Garth Robinson, Kevin Rodgers, Sue Sanvido,
Debra R.  Schneider, Michael Sierchio, Brian Wesley Simmons, Susan B.
Sinnott, Michele Smith, Charlie Sorsby, Thomas A. Spraggins, Orca
Starbuck, Stormwind, Renee Strong, Alison Suggs, Karl Swartz, Laura
Toms, David C. Tuttle, Esa Viitala, Kathy Vineyard, Michael Wagner,
Sky Warrior, Joseph Weinstein, Sara Weinstein, John Wells, Mary Ann
Wells, Kathy Wilk, Stewart Winter, Rusty Wright, and Jeff Young.

Extra thanks to Marla Belzowski, Stephen R. Lee, Liza Miller, Lily
Mummert, Michael Sierchio, Charlie Sorsby, and Rusty Wright for
extensive editorial comments and a wealth of contributions.

----------------
This file is Copyright (c) 1992 by Cindy Tittle Moore.  It may be
freely distributed in its entirety provided that this copyright notice
is not removed.  It may not be sold for profit nor incorporated in
commercial documents without the author's written permission.

Cindy Tittle Moore
Internet: [email]  UUCP: ...!ucbvax!ucivax!tittle
Bitnet  : cltittle@uci        USmail: PO BOX 4188, Irvine CA 92716
----------------