subject: Re: Multi-part fakes: Questions

text:
Thomas <[email]> wrote:

Yes it can be used afterwards, too. However, calculating the parity-data
takes some time. It is probably more convenient to just repost full
segments than to create some par files and sending them.

I honestly can't tell what the exact cause is of random drops within a
series of messages. I only see it now and then. But seeing that happen for
freshly incoming files, I'm tempted to believe that's due to capacity
problems of a link not of a disk.

It increases volume, yes.

No, nothing at all. You can even extend a par-set later, if your initial 5
parts were too few, you can generate additional ones (as long as you still
got the original file(s))

That's actually what RS-Parity does. You don't need to provide the missing
blocks, you need to provide one parity-file which is at least the size of
the gap (more or less, I'm deliberatly explaining it easier than it is) The
very same parity file fixes 20-30 in A and fixes 60-70 in B's data.

That is the beauty of the RS-Raid technology. The disadvantage is: It
requires quite some CPU time to compute the parity data or to fix data with
parity (its not too bad though, a 1ghz machine probably waits longer for
file io to complete) and its definitly far more difficult for a programmer
to implement (instead of a simple repost)

A Parity file must always be as large as the largest file in the set
(looking at how it is used today in usenet) So if your file is splitted as
10x12MB files and one 15MB file, then all parity files will be 15MB in
size.

It will be a bit different in xBin though, for there are no file splits
anymore, only a single file. Still for the RS-Raid system to work, there
must be a blocksize of some sort.

Yes, the Parity parts would be just another source like a repost/fill post,
etc. However for that to work, the parity parts must be connected to the
original file's file_hash, or the reader program wouldn't consider those
parts.

Both can be implemented at the same time. And I rather see both in 1.0 than
creating the need for a 1.1 version.


author: Curt Welch

subject: Re: Multi-part fakes: Questions

text:
Rene <[email]> wrote:

xBin combined with a simple to use fill system (for both semi-automatic
requesting and picking up fills), might cause an endless-post effect.  A
large popular file gets posted once over a period of a week and it never
stops.  For the next 10 years people constantly post and repost fills for
it.  The people that picked up fills this week are the ones posting the
fills next week.

It's probably unlikely that any file would be popular enough to last years,
but it's quite possible that files will re-cycle multiple times before it
stops.

One thing news servers might do down the road is to automatically reject
incoming reposts if the server still had articles in the spool covering
that file segment.  So servers with longer retention wouldn't have to waste
as much disk space on reposts.


author: Kjetil Torgrim Homme

subject: Re: Multi-part fakes: Questions

text:
[Curt Welch]:

nice.

you could also do a "master fill" -- a XOR of all the parts (like the
parity disk in RAID5) which will fill any hole, as long as only one
article is missing.  the size of this article will be the same as the
largest fill, but downloader will now how much of it should be
ignored, since the hole is only one wide.

I think we only need to define how to identify msgids for auxilliary
information (so that simple implementation know to ignore them), then
we can add PAR or other mechanisms at our leisure.


author: Thomas

subject: Re: Multi-part fakes: Questions

text:
Curt Welch wrote:

Also possible would be to use a remember mechanism: I've seen this file
completely, so I'm not goint to pass it through any more - neither repost
requests nor articles.

Probably needs some thinking in the xBin spec to make this easy enough to work.

Thomas


subject: Re: Multi-part fakes: Questions

text:
Kjetil Torgrim Homme <[email]> wrote:

I think the PAR solution I just posted (as comments/modifiations to Rene's
PAR post) does all this and more, so it's better to just drop this idea and
go with PAR.

You could do both under the same system by reserving one special PAR
sequence number (0?) to mean "XOR".  The Reed-Solomon equations would start
at 1 and count up.  But then par file 0 would be of no use with the other
par files. It would only be useful when you only needed to replace one
missing article.  If you needed to replace more than one, then you would
need that many par files with sequence numbers > 1. and the PAR 0 file
would be useless to you.  And if you had the code to generate and use the
Reed-Solomon formulas, there woule be no reason to waste your time using an
XOR formula to generate the par file.


author: Rick Pikul

subject: Re: Why Peter da Silva posts lies about Mancini?

text:
{Followups cleaned, if you want to add them back go ahead it's
just my advice.}

Dr. Michael Mancini ([email]) wrote:
: Peter da Silva ([email]) wrote:
: : In article <[email]>,
: : Head Librarian  <[email]> wrote:
: : > First you say that you're not sure whether he was a doctor or not, and
: : > then in the other article you say that he was _pretending_ to be Houston
: : > doctor. i.e that he _wasn't_ a doctor.

: : Between the first article (the second one you quoted, by the way) and the
: : second (the one you just followed up to) I was sent some email that indicted
: : that the original information I had might be in error.

: Is that a diplomatic way for telling us you're a liar?

	No, it's a simple way of saying that he had more information at
the time he made the second post and that his first post was not entirely
correct, although it was the truth as he knew it at the time.

	Remember:  A false statement is not necessarily a lie, and a lie
is not necessarily false.


author: mrsteveo

subject: Charter Communications NOC Staff -- Please contact me...

text:
Dear Charter DNS Staff,

At some point you used to host 'sosubway.com.'  However, now
InfoStructure (mind.net) hosts that site.

However, for some reason your name servers still point to your servers
instead of mine.

I called your NOC staff late last week on Friday and they said they
would escalate the issue but it didn't sound promising as it sounds
like your hands are tied with 'non-customer' issues.

We really need Charter to remove the stale DNS/mx records so that
other Charter customers can then once again email my customer,
sosubway.com.

We would really appreciate your timely handling of this as it impacts
their ability to conduct business.

Here is what your records show:

root@corolla:~# host sosubway.com 209.225.8.159
Using domain server:
Name: 209.225.8.159
Address: 209.225.8.159#53
Aliases:

sosubway.com has address 64.209.227.103
sosubway.com mail is handled by 10 mail.sosubway.com.
root@corolla:~#

Here is what record(s) should reflect:

root@corolla:~# host sosubway.com
sosubway.com has address 69.9.129.6
sosubway.com mail is handled by 30 Smtp-in-3.userservices.net.
sosubway.com mail is handled by 10 Smtp-in-1.userservices.net.
sosubway.com mail is handled by 20 Smtp-in-2.userservices.net.
root@corolla:~#

Please look into this ASAP.

Regards,

Steve
InfoStructure / Prime Time Ventures, LLC
Support Technician
541-482-8324, Option 1
http://www.mind.net


author: Rich Clark, aka Left Rev Egg Plant, ULC, CotSG

subject: Re: Charter Communications NOC Staff -- Please contact me...

text:
mrsteveo wrote:

I find this post to be similar to my going out on my front porch and
screaming its content into the air.

How do you know Charter NOC staff monitor and respond to messages in
this forum?

Wouldn't picking up the telephone and dialing up the people who could
act on this request be a much more efficient, productive use of your
time and effort?


text:
On Jan 21, 11:14 am, "Rich Clark, aka Left Rev Egg Plant, ULC, CotSG"
<[email]> wrote:

Yeah, it would.  Too bad I tried that and it hasn't worked.

I've emailed abuse, ipaddressing, hostmaster, etc etc.

I've called the phone # on their WHOIS and left VM with how to get a
hold of me.

I called a NOC 800 # only to get told they would fix it; they have
not.

I've tried calling and emailing everyone at Charter I can think of.

I get the same response:

'We only help our customers... not others.'

That's great -- so if you're an ex-hosting customer of theirs and they
don't remove the records, there is no way I can tell or see to get
that issue fixed?  That just can't be true.

So I went on a limb posting here hoping someone either works for
Charter or could pass it on.  I've given the matter all the attention
I can and have emailed and phone called all the different #'s
available to me and I cannot find a single person who will fix it.

I talked to ONE GUY at the NOC 888 # someone gave me who claimed he
saw what was going on and would escalate but he clearly made a
statement that it might not get worked as 'I wasn't a customer.'
Well, the former customer is no longer a customer of theirs either.

I'm at a loss on how to solve this.


author: E-Mail Sent to this address will be added to the BlackLists

text:
mrsteveo wrote:

What does the whois for the domain point to?

 Checking server [whois.tucows.com]
  Domain name: SOSUBWAY.COM
  Domain servers in listed order:
   NS2.MIND.NET
   NS.MIND.NET

 Is Charter your domain DNS provider?
  {Seems not.}

  If not, would that not only affect their customers?

   As the rest of the world will be looking at the DNS
     the whois records refer to.

    SOA: NS: ns.mind.net
     NS: ns2.mind.net
     NS: ns.mind.net

      sosubway.com -> 69.9.129.6 -> mentat.mind.net
  www.sosubway.com -> CNAME: mentat.mind.net -> 69.9.129.6 ->
  MX: Smtp-in-1.userservices.net -> 204.16.46.1 -> smtp-in-1.userservices.net
  MX: Smtp-in-2.userservices.net -> 204.16.46.5 -> smtpscan-4.userservices.net
  MX: Smtp-in-3.userservices.net -> 204.16.46.4 -> smtpscan-3.userservices.net

      CIDR: 69.9.128.0/19
  NetRange: 69.9.128.0 - 69.9.159.255
   NetName: INFOSTRUCTURE-NET-1
   NetType: Direct Allocation
   OrgName: InfoStructure
NameServer: NS.MIND.NET
NameServer: NS.IOR.COM

      CIDR: 204.16.40.0/21
  NetRange: 204.16.40.0 - 204.16.47.255
   NetName: VISPTECH-ARIN-DIRECT-1
   NetType: Direct Allocation
   OrgName: VISP Technologies
NameServer: NS.VISP.NET
NameServer: NS2.VISP.NET

I don't see how it affects anyone except Charter customers.

 If it does affect one of those Charter customers,
  that customer could take it up with Charter,
  and point out that their DNS is serving stale records,
  instead of getting the NS from the whois.


author: Martijn Lievaart

text:
On Mon, 21 Jan 2008 11:47:06 -0800, mrsteveo wrote:

Easy. Let a customer who cannot email you call Charter.

/Ducks and runs.

M4


text:
On Jan 21, 1:28 pm, E-Mail Sent to this address will be added to the
BlackLists <[email]> wrote:

You're right.  It didn't affect anyone but Charter folks.

On the other hand... someone from Charter finally called me and said
they took care of the issue.

I tested it -- and they did take care of it.

FYI -- Someone at Charter Business Services -- 888-254-3534.  So
apparently that # will get you help and even a return phone call if
you're squeaky enough.


text:
On Jan 21, 4:12 pm, Steve Baker <[email]> wrote:

I'm by no means a DNS expert although what I do know is Charter
customers could not email sosubway.com after sosubway.com changed
hosting from Charter to InfoStructure.  That I know to be fact.

For whatever reason, when you queried any name servers other than
Charters; you got a correct response.

When you queried Charters name servers, you got invalid responses.
This also lead to Charter customers being unable to email the
sosubway.com domain successfully.

After Charter called me and said they 'fixed' something I checked
against a name server of theirs and it still showed broken.  However,
what domain name servers they give their cable modem subscribers (and
I don't know what they are..) must be fixed because Charter users can
now email into sosubway.com without issue so something was fixed.

That's what I do know to be fact.

I'm by no means a network, DNS, or MX record guru by any means and I
really appreciate(d) the input provided here.  Valuable learning
experience and I hope what I said makes sense -- sorry I am so naive
on the subject. :(

Thanks.


author: stinky

text:
In article <[email]>,
 "Rich Clark, aka Left Rev Egg Plant, ULC, CotSG"
 <[email]> wrote:

If Charter NOC staff does monitor this newsgroup I would request that
they add ALL their dynamic/residential IP ranges to the Spamhaus PBL
lists.


text:
mrsteveo wrote:

Well, right there's the focus of your attention. Squeaky wheel gets the
grease; keep calling until it is fixed.

It might get fixed if you continue to address the problem to the proper
channel. Apparently you've not succeeded in finding the proper channel
or haven't become a large enough pain in the ass to people you've found
thus far. Maybe a concise, strongly worded complaint via snail mail or
fax would be more convincing?

Again, probably as effective as standing on your own front porch and
screaming at the wind.

There's another one; keep at this guy until he gets his head out of his
bum and makes good on his promise.

Charter are well-known for these kinds of hijinks. You've done more than
enough, far more than I would have. Posting about it here seems to be an
extension of your wasted time.


author: NFN Smith

text:
mrsteveo wrote:

Have you tried going through Charter's sales people?

I've found that in certain cases, a sales rep can sometimes be the most
effective customer advocate, when you're dealing with the big companies
where internal communication lines seem to be broken.  If the company
has enough focus on sales/marketing, the sales people sometimes have
more leverage in trying to get things done.

This may not help with Charter, but it's worth a try.

If you can't get any further, then about your only option is to take
your business to another provider, and make sure their sales people know it.

Smith


author: Sam

text:
mrsteveo writes:

Try having your lawyer draft a polite letter to Charter asking them to cease
and desist from hijacking a domain they are no longer authorized to host.


author: Steve Baker

text:
On Mon, 21 Jan 2008 21:28:37 GMT, E-Mail Sent to this address will be
added to the BlackLists <[email]>
wrote:

  Well, the DNS system doesn't use whois records; whois is supposed to
reflect the actual DNS situation, but it doesn't always.
  The only way this situation of old records still living in the
Charter servers would affect anyone is if the authoritative Charter
DNS servers were being used as caching/recursive servers by Charter
users, which seems unlikely.


text:
On Mon, 21 Jan 2008 15:06:15 -0800 (PST), mrsteveo
<[email]> wrote:

  Their nameservers still answer authoritatively for the domain
sosubway.com. Was Charter responsible for updating the NS records of
sosubway.com in the parent DNS servers of the .com domain? If not,
this problem had nothing to do with Charter. If the DNS system doesn't
point to a nameserver as being authoritative for a domain, it doesn't
matter what that nameserver has to say about the domain.


author: Carl Byington

text:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 21 Jan 2008 18:34:50 -0800, mrsteveo wrote:

Well, not really:

dig sosubway.com ns @209.225.8.159

;; ANSWER SECTION:
sosubway.com. 10800   IN      NS      auth0.ns.charter.net.
sosubway.com. 10800   IN      NS      auth1.ns.charter.net.

;; ADDITIONAL SECTION:
auth0.ns.charter.net.   3600    IN      A       209.225.8.159
auth1.ns.charter.net.   3600    IN      A       209.225.8.160

However, you have another handle on them:

Domain name: SOSUBWAY.COM

 Administrative Contact:
    Hoxley, Jeff  [email]

Given that the administrative contact for that domain seems to be a
customer of charter, Jeff should have standing in Charter's eyes to
complain about this and expect to get it fixed. Those expectations of
service on Jeff's part may have nothing to do with the actual service
delivered by Charter.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFHlYbjL6j7milTFsERAqAMAJ939HcDTOJCxHpQOW3IKASB51zu3ACePjeK
XWIB/Li4F3RIxNRghIubem4=
=qWOn
-----END PGP SIGNATURE-----


text:
On Mon, 21 Jan 2008 18:34:50 -0800 (PST), mrsteveo
<[email]> wrote:

  All this does shed a different light on things. It's a puzzler.

  Just to clarify... that started on Friday? Were those "any name
servers" servers that would be expected to have info about
sosubway.com cached? The current TTL for the MX record in the Charter
authoritative servers is 10800 (3 hours). Do you know if that has
changed since Friday? Just wondering if there was previously a very
long TTL that could account for things.

  Or a cached TTL finally expired so the new DNS records came into
play. Are you sure Charter users were the only senders that were
having a problem? What I'm saying seems pretty unlikely to me, I'm
just kinda thinking out loud.

  Maybe Charter had a setup where the DNS servers the users used were
updated with the records from the authoritative Charter servers. Hmm,
that kinda makes sense. It might cut down on bandwidth use and
increase the speed of lookups, and it's something that would need
fixing. Maybe it was an artifact of a time when the authoritative
Charter servers were authoritative only for the Charter domains. And
maybe that's one of the most ridiculous wild assed guesses I've ever
made. ;-)


text:
NFN Smith wrote:

Isn't that what he already did, which is why Charter won't
 do anything about the issue?


text:
Steve Baker wrote:

The registrars don't use the authoritative name servers
 that they store in their whois database for changes to
 the TLD zone file?

 Are there any registrars that provide for changes to the
  TLD zone file, for a change in a domain's authoritative
  name servers, without changing the authoritative name
  servers in the whois database?


text:
On Tue, 22 Jan 2008 04:57:42 GMT, E-Mail Sent to this address will be
added to the BlackLists <[email]>
wrote:

  Updating whois records and updating NS records are two different
things, so I'd say the answer is no. I think the NS records in the
parent servers are generally updated more often than the whois
records.

  I've seen registrars suspend domains (remove NS records, or
substitute "ns.suspended...com", for the domain in the parent DNS
servers), but still leave the old DNS servers in the whois info. Check
out loanofficerassociation.net, for example. But in general the whois
records do get updated to reflect the actual DNS situation, but the
whois record updates can lag behind the DNS system updates, so the
whois info isn't always accurate. It used to be the opposite, that the
DNS system updates would lag behind the whois updates. Of course, I'm
just generalizing, it all depends on the registrar and registry
policies, but I think a registrant can get the NS records in the DNS
system changed in just a few minutes for a .com domain.


author: librarian

subject: Re: [RFD] rmgroup news.admin.net-abuse.policy

text:
Well, since news.admin.net-abuse.policy is censored,
and censored to the hilt, not even allowing ANY
crossposts to any other groups, this post will be
reposted to different groups separately.

Hey, peverts, grep that spool now.

In article <[email]>,
[email] wrote:

Yes, the armory is indeed "official".

Now, can I ask you a kwestion:

WHO made you "official" here?

Have you been appointed by the court?
Elected by usenet as a whole to represent
the interests of da people?

Or you are but a self-appointed, intolerant,
blood boiling censor
and totalitarian dicator wannabe?

Can you produce a cerficiate,
cerifying you as an "official"?

Why don't you unplug all the idiot boxes in the world,
you call TV?

What are they full of?

Violence of horrendous proportions?

Well, well, well.

Why are you wasting all your "official" capacities here then?

Go cleanup the "entertainment" industry first.

Start with Ted Turner first.

Oi, oi, oi, oi, oi.

Oh, capitalizing on "suespam" issue?

VERY impressive indeed.


author: Edmond H. Wollmann

subject: Re: Does belief in astrology cause insanity? An academic review

text:
On May 31, 7:43 am, [email] (Closed Caption) wrote:

Off topic.

http://www.astroconsulting.com/FAQs/charter.htm

Edmond H. Wollmann P.M.A.F.A.
© 2010 Altair Publications, SAN 299-5603
Astrological Consulting http://www.astroconsulting.com/
Artworks http://www.astroconsulting.com/SDSU
http://home.earthlink.net/~arcturianone/


author: Cujo DeSockpuppet

text:
"Edmond H. Wollmann" <[email]> wrote in news:3a7b3904-
[email]:

The OOoKaT previously ruled it was on topic. Fuck off, Pantyhead.


subject: I'm a guy, but I got girl issues?

text:
You REEALY REEEALY don't want to make me angry. I might give you a
verbal outburst.


author: DCOM

subject: Expired Posts?

text:
I'm using Xnews 5.04.25. How do I set it up so that it doesn't display
posts that are no longer on the server? I have set storage option under
"setup" to = 0 days which is supposed to purge posts when they are no
longer on the server. That doesn't seem to work as when I click on some
posts it asks if I want to use Google.


author: Tech Zero

subject: Re: Expired Posts?

text:
The voice of "DCOM" drifted in on the cyber-winds,
from the sea of virtual chaos...

You don't... When using Header storage "expired" & "older then" posts
are removed when a group is close. As for other messages, ones that are
missing because of cancels and such, blame them on the server for not
removing them from the overview fast enough.


author: MartinS

subject: Re: Expired Posts?

text:
DCOM <[email]> wrote:

That's likely a server problem, not an Xnews problem.


author: Gerry

subject: Slow getting headers of all messages in group

text:
I'm making an NNTP client and I am able to have it fetch all the headers
from a newsgroup to produce a list of the messages for the user to
choose from.

However, my program seems several times slower than others at fetching
the headers of all the articles in a group.

What is the best algorithm to do this?

I've tried repeatedly sending 'next' and 'head' until I reach the last
message.
And I've tried indexing from the first message number to the last
message number sending 'head <message number>'.

Both methods work fine, but they are not nearly as fast as other
newsreaders.


author: Curt Welch

subject: Re: Slow getting headers of all messages in group

text:
[email] (Gerry) wrote:

If you pipeline the commands you will do much better.  But the code is
harder to write.  By "pipeline", I mean send commands without waiting for
the reply before you send the next command.  But that's tricky stuff to get
right.

They use XOVER, not HEAD.  XOVER is a bulk dump of only the important
headers from all articles in the range.


author: Russ Allbery

text:
Gerry <[email]> writes:

This is what overview was designed for.  See the XOVER command (described
briefly in RFC 2980).


text:
Curt Welch <[email]> wrote:

Thanks for the replies.

I noticed XOVER, but also saw that it is an extension to the original
NNTP protocol.

I avoided it because I wanted to be sure that my client would be
compatible with all NNTP servers it encountered.

Is it pretty safe to assume that any news server I connect to will
support these extended commands?


text:
In article <[email]>, Russ Allbery <[email]> wrote:

Zig heil!

A few quotes for a warm-up cycle...

"Beware the leader who bangs the drums of war in order
to whip the citizenry into a patriotic fervor, for
patriotism is indeed a double-edged sword.

It both emboldens the blood, just as it narrows the mind.
And when the drums of war have reached a fever pitch and
the blood boils with hate and the mind has closed, the leader
will have no need in seizing the rights of the citizenry.
Rather, the citizenry, infused with fear and blinded by
patriotism, will offer up all of their rights unto the
leader and gladly so.

How do I know? For this is what I have done. And I am Caesar."
- Julius Caesar

In 1919 Joseph Schumpteter described ancient Rome in a
way that sounds eerily like the United States in 2002.

"There was no corner of the known world where some
interest was not alleged to be in danger or under actual
attack. If the interests were not Roman, they were those
of Rome's allies; and if Rome had no allies, the allies
would be invented. When it was utterly impossible to
contrive such an interest -- why, then it was the national
honor that had been insulted. The fight was always
invested with an aura of legality. Rome was always being
attacked by evil-minded neighbours...The whole world was
pervaded by a host of enemies, it was manifestly Rome's
duty to guard against their indubitably aggressive designs."

Herman Goering, president of the Reichstag,
Nazi Party, and Luftwaffe Commander in Chief:

"Naturally the common people don't want war: Neither in
Russia, nor in England, nor for that matter in Germany.
That is understood. But, after all, it is the leaders of the
country who determine the policy and it is always a simple
matter to drag the people along, whether it is a
democracy, or a fascist dictatorship, or a parliament,
or a communist dictatorship.

Voice or no voice, the people can always be brought to
the bidding of the leaders. That is easy. All you have
to do is tell them they are being attacked, and denounce
the peacemakers for lack of patriotism and exposing the
country to danger. It works the same in any country."

- Herman Goering (second in command to Adolf Hitler)
  at the Nuremberg Trials

"The whole aim of practical politics is to keep the
populace alarmed (and hence clamorous to be led to safety)
by an endless series of hobgoblins."
- H.L. Mencken

Now...

This is about the corrupt "power elite" of usenet,
the same thing as your corrupt governments
and about those evilest of all evil Freemasons,
who rule your sorry asses,
interested in maximizing the process
of sucking the blood of many by the few.

It is about your sick world and the WWIII,
you have cooking on your plate.

It is about bio-robotization of humans
and reduction of human intelligence
to the level of a functioning machine.

It is about your totalitarian dictators
and fascists, manipulating your guilt
and fear, while posing as some kind of
public servants, protecting all from
"evil", they themselves invented and
defined.

It is about the evilest of ALL evil,
calling the others evil, just as a decoy.

It is about corruption and deceit of
proportions unseen.

It is about so called New World Order,
and the foundations of it.

It is about these vicious criminals
against humanity, destroying your freedoms,
while making the sucking sounds with their
mealy mouths about protecting those very freedoms.

It is about the hunt down on your dissidents,
the best of your kind.

It is about you, people of the planet Earth,
during the last convulsions
of the Age of Corruption,
you all being fed as Information Age.

It is about your only two remaining "laws":

1. Money = god,
   and god = money.

2. Either I suck your blood,
   or you will suck mine.
   There is just no other way.

It is about your world, coming down on you.

Eat it now.

In article <[email]>, piranha <[email]>
wrote:

Oh, finally, long time no see.

Ladies, Gents and assorted bio-robots,
itshehebe piranha, the big-sucking-8 fuehrer wannabe is here
to continue the campaign
of global deception and lies,
of fabrication and perversion
of all there is to pervert.

The first step, usually, on the way of self-promotion,
is licking asses of those, who are long due to retire,
and yet unable to loose their grip on power.

Yes, "power corrupts and absolute power corrupts absolutely",
and usenet gives these fascists an equivalent
of absolute power.

Zo...

Licking the ass of Russ Allbery REALLY good
is about the best way to be noticed
and have a chance to be appointed,
mind you, APPOINTED
by the current clique of usenet fascists,
ruling the big-sucking-8,
the most censored, most blatantly fascist
and yet, most propagated hierarchy there is.

It is a hierarchy where the most blatant
censorship and group takeover with perverted
tricks of "moderation" by the most blatant,
self-admitted fascists, such as David Kinny,
who took over one of the most important groups,
the root of the entire AI hierarchy, comp.ai,
who conducted a campaign of lies, fabrication
and outright deceit,
is not only accepted, but promoted and cherished.

First of all, power on usenet,
like in the fascist or totalitarian systems,
is passed via appointment.

Screw the idea of democracy.
That idea is outdated and cancelled,
just as the 1972 global nuclear self defense treaty
cancelled by this 2nd derivative Freemason, Bush,
that senile monster with a mind of a juvenile,
who came to finish the trip of New World Order,
as blatantly and arrogantly proclaimed
by his own father,
one of the slimiest CIA figures in history,
whose dumb ass was ruled by the "silver fox", his wife,
just as was the case with the rest of them,
"rulers" throughout the times.

The same story as with that honorary 33 degree Freemason,
Reagan, who used a decoy of calling Russia the "evil empire",
just to delude all and distract everyone's attention,
plus put Russia on defensive.

"The best defense is an attack",
an ancient Roman technique,
and that 3rd rate movie actor
with mentality of a deranged cowboy
knew it all too well.

Because the REAL evil empire was his,
and he knew it all too well.
Service to that very evil empire
was exactly the reason he was awarded
a honorary highest degree in Freemasonry,
for his "long time and extensive cooperation
with Freemasons".

The question naturally arises:
What kind of "cooperation"
and what kind of "projects", significant enough
to result in awarding him the highest degree
in Freemasonry, were carried out by him
to benefit the agenda of Freemasonry?

Another words, a conspiracy behind the scenes
between Freemasons and Reagan
that resulted in some of the most devastating
developments in the world affairs,
directed by the hands of those servants of the beast,
whose sign is 666,
posing as some kind of seekers of light,
heart and truth just to delude all
as they always do.

On every single bar code
there is a sign of the beast.
The leftmost two thin lines are 6
and so are the rightmost two lines
and so are the two lines in the middle,
thus implementing the commandment
of that very beast:

"On the forehead of every single man
or in his right hand,
there must be my symbol, 666"

They seek power over others,
forever improving their skills of manipulation
of symbology with the help of occult and Kabbala,
during their secret rituals deep at night.

Some references before we go on:

http://educate-yourself.org/mcsvaliinterviewpt1.html
http://www.freemasonwatch.freepress-freespeech.com/
http://www.parascope.com/articles/0997/skullbones.htm
http://www.hiscorearcade.com/skullandbones.htm
http://www.secretsofthetomb.com/excerpt.php
http://luxefaire.com/sculland.htm
http://www.knightstemplar.org/
http://hem.passagen.se/thebee/EU/global.htm
http://www.sfmoma.org/espace/rsub/project/disinfo/prop_newordr_trilateral.html
http://www.angelfire.com/co/COMMONSENSE/armageddon.html
http://www.angelfire.com/co/COMMONSENSE/wakeup.html
http://pnews.org/boards/feminism/messages/6.html

Now...

Reagan invented the idea of star wars,
trying to create the biggest military advantage
for his evil empire
in order to complete the trip of global domination,
just as Freemasons and Illuminati crave for
as evidenced on the back of one dollar bill.
"Novus Ordo Seclorum" - "New World Order".
That is what this is all about.

Yes, the same as here, control, domination, oppression,
violence of unlimited and unprecedented grade
and things of that nature. The same perverted,
crimially minded members of the "power elite",
hiding behind the concepts of democracy, freedom
and "protecting the american people form 'evil'",
they themselves invented and defined,
just like it is here, on big-sucking-8.

The same tricks, the same lies,
the same manipulation,
the same mass deceit,
the same violence,
the same blood boiling idiocy and intolerance.
The same totalitarianism and fascism
of raging grade, knowing no limits whatsoever.
Another words, the same old shit
since the times of poisoning of Socrates,
who gave you all the very idea of democracy,
about the most advanced idea
in the entire history of mankind
besides the idea of love,
being the very essence of life,
given to you by Jesus.

You nailed that one to a piece of wood also,
just as you always do to anyone, who stands
on your rotten, corrupt and violent way.

Then you have built the churches in his name
and used him as some kind of a commodity,
pretending to be serving him,
while, at the same time, stuffing your bellies,
making everyone pay you a "sin" tax,
using the tricks of guilt and fear manipulation.

Fear
and guilt.

That is how you tick by now
as a result of unending manipulation and
mind programming by the evillest of all evil,
manipulating your subconscious every place you look.

The very concept of god was perverted,
as though god was some kind of a ferocious monster
to torture you all
for non submittal to the doctrine,
even though that very Jesus told you
"how can you punish the blind for not
being able to see".

You burned Pythagoras alive and with him
the biggest library and accumulation of
the most advanced knowledge
on the face of the planet Earth,
he gathered traveling by foot
throughout all lands,
that was said to be burning for months.

Pythagoras gave you the most essential ideas
you use in modern technology and science,
the ideas that preceded your most advanced
scientific work by nearly 2 millenniums.
The Furrier analysis is but a byproduct of his
ideas. The idea of an octave and a concept of
harmonics, that is at the core of modern science,
technology and music was his idea.

The ideas of the most profound and mystical
significance of sacred cosmic proportions were his.

The solution of a triangle that finally laid
the foundation of modern civilization
as, for the very least, it allowed precise
calculation while dividing land into parcels
was created by him also. It helped to resolve
many conflicts between the people
and increased the precision of architecture
and science by the orders of magnitude.

The ideas of Pythagoras lay at the very foundation
of Masonic obsession with symbology to a large
extent. They took that sacred knowledge
and eventually perverted it to the point
well beyond horrendous,
finally leading to what you have now and,
specifically, the idea of New World Order.

Freemasons, the direct descendants of Zealots,
used the ideas of Pythagoras and converted it all
into a service of the beast, whose sign is 666.

Now, what remains of Pythagoras
"could comfortably fit on a post card",
so utterly it was all destroyed,
just to prevent any genuine progress
and a process of de-hypnotization of the masses,
kept in all sorts of delusions
throughout the times by the same evil,
blood thirsty conmen,
who are so negative and so uncreative
that the only thing they are capable of
is to forever invent new ways
of sucking the blood of others
and enlarge their bellies,
driven by the fear of "survival"
as they know not that very trust,
that propels all life.

Finally, they came to the point
where the real "god" of theirs
is the paper good of money,
and so they ended up forever
accumulating it, hoping to somehow
manage to run away from that very fear
of "survival".
They even invented this perverted idea
of "survival of the fittest",
the most devastating and life condemnating idea
there is.
That is the extent of their "creativity".

It is such a profound negativity
and mistrust toward their own being
and all life as such,
that it basically invalidates
the very concept of divine or "god".
If sucking blood of many by the few
is what you call "creativity",
then what your very idea of god is worth?
What is this very craving and thirst
for blood of others if there
is god to guide you all
and provide you everything in nature and beyond?

It is UTTER and COMPLETE negation of life.
It is an escapism from creativity
and brotherhood of all people
of the planet Earth
and not only people,
but all life as such.

Only if you TOTALLY mistrust
that very essence of life
can you become such blood thirsty monsters,
seeking one thing and one thing only,
maximization of the rate of sucking
of the blood of many by the few.

You massacred Mansur,
and you annihilated so many
of your brightest starts,
that it is simply amazing
you survived to this day.

You forever try to kill every single one,
who challenges your thirst
for sucking the blood from many
by the very few.

As a result,
you have not made any genuine progress
since the stone ages.

Now, it is OBVIOUSLY taken for granted,
and undisputedly so,
that Russ is on the par
with the world caliber "heroes",
better known to the masters throughout times,
as the biggest and most destructive monsters
that ever walked the face of the planet Earth.

Let us analyze it and see what those "heroes"
are made out of.

Now...

This conman Bush,
that mentally retarded idiot
with mentality of a 2 year old,
utterly incapable of grasping
the consequences of his own actions,
and can not even be held liable
as a criminal against the mankind
on the basis of his utter mental inability
to grasp it all,

without ANY proper legal proceedings,
without ANY evidence,
accepted by the courts of proper authority,
started a war,
and not against some terrorists,
but against the country of Afghanistan
merely furthering the agenda of NWO
and global domination.
At this junction, it is totally controlled
by the evil empire.

His initial claim
that they are interested in finding
"those, responsible for this horrible act",
was pure deceit,
just as you see it here, on usenet.
His real interest is the world hegemony
of evil, as the concept of New World Order,
pronounced by his own father states.
It is global domination, fascism and totalitarianism.
It is converting the people into puppets
and bio-robots, the functioning machines,
used by these very monsters
to become fatter and fatter and fatter
in their never ending thirst for blood.

That is why they created an extensive
collection of movies on the subject
of vampirism. It is done in order to create
the subconscious acceptance of vampirism
and make your vulnerable, suggestive
and programmable minds accept that very
"power" as something real,
planting the subconscious fear in you all.
Fear on one side and inevitability on the
other side. Cunning as it gets.
As Freemasons know all too well,
once you accept ANY idea,
you become a subject to it.
You become vulnerable to symbolic manipulations
using it as a sword of sorts.

That 3rd derivative Freemason, Bush,
knows ALL too well
that about the ONLY card in his deck
is the card with the sword.

Otherwise,
he can not possibly be of any assistance
to his own country
or solve ANY of the most urgent problems,
facing the world,
coming down on you.

His intelligence is on the par with an amoeba
and about the only thing he is interested in
is a fascist boot of "might makes right".
That is his only and exclusive tool and weapon.

The sword is the ONLY possible way
for him to keep his "ratings"
and look like something real,
something of significance.

He will create some the most devastating events
in the entire history of mankind
while perverting it to look like
he is a genuine servant of "good",
ridding the world of all "evil",
even though he is the manifestation
of that very evil,
spreading the fire of violence
throughout all lands
and achieving his own agenda
of world domination
manipulating and fabricating
left and right.
A typical mind of an evil monster
from the lands of the Masonic empire
of the united sucking states that
blatantly and arrogantly put the very
evidence of its satanic origins
on its own currency.

He poses as true and genuine servant
of his own people,
which is a lie, and nothing but a lie,
as he is the servant of the beast,
the one that sucks your energy,
interested in creating
the world hegemony of evil.

The evidence is all over.
It is so clear and concise,
that you must be totally deranged
not to see it,
or be such a pathological case,
manipulating the most perverted lies
to make them look like truth,
that it is well beyond comprehension.

The whole operation in Afghanistan
is a scam
and it is not clear,
and will probably remain secret
until the end of times,
who did it and why
and what agenda does it serve.

Using this scam with Afghanistan,
these evil monsters were able to
manipulate the world opinion,
playing the same guilt and fear card,
and obtain an overwhelming approval and
support and, at the same time,
boost their military budget to the levels
exceeding the cold war period by far.

Why?
For what?

These very servants of evil run their mealy mouths
about "cold war is over" and
"a new deal and cooperation" and,
at the same time,
spending more money on the military
than the entire Europe and Japan combined,
all in the name of "fighting terrorism",
being the masters of manipulation and fabrication,
using the most potent tricks
of manipulation of symbology
learned from Freemasons and Illuminati.

They are spending $400+ billions annually on this scam,
which is factors of magnitude more than
it would take to restore the entire economies
of Eastern Europe, Africa, Afghanistan and
the entire Middle East combined.
But they are not interested in restoration
of those economies. Instead, they are interested
in finding new tricks to facilitate their agenda
of global domination, keeping those economies
in a devastated state.

They brag about "free trade" and yet, keep the entire
world in powerty, starvation and misery just to make
sure no one can match their race of "blue blooded"
and they remain the exclusive dictators on the face
of the planet Earth.

The budget of this size is factors of magnitude more
than it would take to rid the entire world of hunger
according to the opinions of the experts.

The most amazing thing is that they managed
to manipulate guilt and fear of all their
people and support this biggest military
expansion on record even though the former
"enemies" are now being called friends.
How could the military budget of this magnitude
be justified with this "war on terrorism" scam?
Even at the times of the most expansive, nuclear
accumulation, the military budget was smaller
than it is now.

They will never "win" the war on terrorism
with these tricks and they are not even interested
in it and they know all too well, it is simply impossible.
Just the other way around.
The more they go around the world
and step on the other people with their
fascist boot of "might makes right",
the more they saw the seeds of that very
terrorism. That is probably the very
strategy of this global deceit as it
begins to feed on itself. It is likely
that the terrorist activity will increase
throughout the world and new countries
will be placed on a "black list" to be hit next
in this world hegemony of evil,
hiding behind the concept of
protecting all from "evil",
just as it is here on usenet.

That is about the fastest way to expand
the domain of the evil empire globally
and no country in the world
would be able to object.
Otherwise, they will be classified
as supporters of "terrorism" and become subject
of the next high tech military attack themeselves.
Cunning as it gets.

Even to believe this military expansion scam
is done in the name of "war on terrorism",
one must be simply senile or naive to the point
of oblivion.

It is not about terrorism.
It is essentially the same thing
as "war on crime" scam. The more war
on crime there is, the more you become
a fascist and a totalitarian state,
creating the very motivation to commit
that very crime you claim to be fighting
just to oppose you.
It is but a scam with the hidden agenda
of totalitarianism and fascism,
meant to convert the people
into functioning machines
eventually to be used
by the fattest parasites there are
to become fatter and fatter,
and build the tremendous enterprises,
feeding on this thing.

It is about the world hegemony of evil
and undisputable domination
by the evil empire
over all lands.

It is not a coincidence
that after becoming a president,
Bush immediately appointed the most ferocious,
the most senile and the most violent monsters
into top positions in the government,
related to violence, domination and military.

The unprecedented process of disassembling
the very foundation of his own country
and destruction of the most fundamental ideas,
such as democracy and justice, constitution
and other most fundamental principles,
has ALREADY been largely completed.


text:
[email] (Gerry) wrote:

The RFCs don't do a very good job of documenting the real world.  People
are working on that problem.  I don't know which RFC you saw XOVER in, and
what other extended commands you are talking about so I can't say if they
are well supported or not.  XOVER however, is very very well supported.
You would be hard pressed to find a server which didn't support it.  Most
newsreaders not only use it, but I bet they don't even implement a back-up
system (like NEXT/HEAD) to deal with the problem if XOVER doesn't work.

What you should do is make sure your newsreader produces a good error
message if it finds a server which doesn't support XOVER (you will get an
5xx code back when you try to use it) so the fact can be sent back to you
(the author of the code) if somebody one day found such a server.  But
unless you have reason to do otherwise, I wouldn't bother with anything
other than XOVER for your first version.  If you want to write a lot of
code which may never get used, then you can drop back to using NEXT/HEAD if
the server doesn't support XOVER.


author: Jeffrey M. Vinocur

text:
In article <[email]>,
Gerry <[email]> wrote:

There are pretty much no safe assumptions when dealing with NNTP.
But if you want to get good performance, you have to take
advantage of XOVER -- and if it's not, fall back to something
else or just display an error to the user.

I don't know the prevalence of support for that particular
extension, but I'd hope it's fairly high.  (Higher than, for
example, NEWNEWS.)


text:
Curt Welch <[email]> wrote:

Thanks for the info.
Seeing as I've already got the NEXT/HEAD method implemented, it wouldn't
be very difficult to have it fall back to it if XOVER fails.

I initially only worked only with the original RFC 977 located here:
http://www.ietf.org/rfc/rfc0977.txt

But then I found another page which explained things slightly better and
was in hypertext. It also describes several extensions to NNTP.
Located here:
http://www.mibsoftware.com/userkt/0099.htm


text:
In article <20030508004745.617$[email]>,
Curt Welch <[email]> wrote:

Actually, MT-Newswatcher has ~"Use XOVER to retrieve headers" as
a checkbox in the optimizations pane, so I assume it has support
for not using it.


text:
In article <[email]>,
Gerry <[email]> wrote:

You should use, instead, the draft version of the forthcoming
replacement for RFC 977.  The latest is:

http://www.ietf.org/internet-drafts/draft-ietf-nntpext-base-18.txt


text:
[email] (Gerry) wrote:

Yeah, that's so old and out of date as to be almost useless.  It's a good
place to start however to understand the basic of what NNTP is all about
and most of what it documented in that is still valid even if the
documentation is lacking.

Many of those extensions are not safe to use - they will work only on a
very limited set of servers.  But most the ones that won't work are also
once you probably aren't even interested in.

I don't think I've ever seen a good document that trys to explain what
actually works and what doesn't work and why.

XHDR and XOVER are very common.  I'm not aware of any "real" servers that
fail to support those.  XPAT is supported in most servers as well, but it's
not as safe to assume it will be available.  It might be disabled even on
some servers which have support for it because of performance issues (users
can put a large load on the server with XPAT).

LIST ACTIVE is supported everywhere.  It extends the LIST command by
allowing an wildmat argument to limit the output.

LISTGROUP is probably supported everywhere but because the command doesn't
have any type of range arguments, it's mostly useless.

MODE READER is something you should do at the beginning of every session
and just ignore the return code.  There are INN servers which, depending on
how they are configured, need this to put the connection in the right mode.
It's a noop on most servers.  But here's something I'm not sure about.
Should you do a MODE READER before or after authentication?

That's about as far as most newsraders will go in terms of trying to use
extra commands if they are trying to work on a wide range of servers.

Mostly with NNTP programming, you have to write your code and then just
test it on a lot of servers to find out if it really works or not and if
any of the servers behave differently and break your code.


text:
[email] (Jeffrey M. Vinocur) wrote:

Yeah, that's a much better RFC.  It does a much better job of correctly
documenting return codes and everything about NNTP.  But, be careful,
because it documents what NNTP "should" be and not what it currently "is"
in a few places.  It for example claims there's a command called OVER and
HDR which are standard extentions and doesn't mention XOVER or XHDR.
That's because they are attempting to convert XOVER and XHDR which
technically have never been a standard and never can be because they begin
with X, into a true standard.  Most servers today however do not support
"OVER" or "HDR" and maybe none do as currently documented in that draft.
It's very likely however that that draft will become the standard (after
more adjustments) so it's most likely where NNTP is headed.  It's a good
reference for anyone writting NNTP code today.

Another good set of usenet standards and links to standards can be found at
the USEFOR site:

  http://www.landfield.com/usefor/


author: [redacted]

subject: Andrew Gierth will censor this valid article          1434404886

text:
Comp.ai is not a personal property of David Kinny,
the censor and self admitted fascist, asserting a superiority
of his own race over Russian and creating black lists
and white lists.

It used to be a group for uncensored public discussions
until he conducted a campaign behind the scenes
to take over the entire comp.ai hierarcy and fabricated
hundreds of votes. None of these "voters" participated
in that group ever since.

The usenet power elite refused to act opon this ugly
takeover and ignored all the facts of the matter,
including the specific quotes by David Kinny himself.

David Lawrence, aka tale has this rule:

YEE SHALT NEVER EVER EVEN CONCEIVE THE IDEA
OF TOPPLING A SITTING DICTATOR, AKA "MODERATOR".

Russ Allbery, aka The Godfather Of Brainwashing
wrote all sorts of excuses to sabotage the news group
creation process.

These perverts and the new totalitarian clique
are only interested in thought control, oppression
and domination and not FACILITATION of discussions
on usenet.

They are power hungry megalomaniacs and totalitarian
dictator wannabes, right this very moment inventing
a new set of delusions they call the "survival guidelines".

What is this, some kind of a warfare here on usenet,
or just a place to excersize you RIGHT to express
your ideas as provided by the Basic Human Rights
conventions, such as Freedom of Speech and
Freedom of Association, and Freedom to Choose Ones
Own Convictions.

Stop this horrendous abuse and return this group
to uncensored state as it was before this takeover.

Andrew Gierth ([email]) is a censor,
mass cancelling valid articles in bulk, supporting this
totalitarianism and engaging in massive destruciton
of the articles.

He is one of the most arrogant self-appointed perverts,
destoying everything he likes while pretending to be
a protector of usenet.

The same self-appointer information terrorist
as Howard Knight, Chris Lewis, Cosmo Roadkill
and others as evidenced on control.cancel group.

These people and totalitarian clique of self-appointed
usenet rulers are the biggest problem on usenet,
suffocating it to death.

They say "if you can cancel, so can I".

I hhdygss ihrdr tnomnapv oypnee nerucndg oormoieur plt
aski satrwpon tifhn tef eh fpnddeet tde in tprqdsai oaa?

Oesei aeer alsopu eteon etcq ieo ssot
prw rcla nmai er rnp ou at ngp eimg pt.

Pvttt niant locp suenr soe mwfktp ztcrmrr iro
siir ttei ehttlfea ebnmes oil idipalu vtphnhamd io
ongo ivusenw ihrrrr toayios ntvtrd ntbnt ndtnn
kcpen rnu acn o ettrnop ett seeee eyips.

Yner iae ioi tfe unfh hol rrcrt ama
elwi iond udpyn itogs ahtv fdeor tfie o rbcdo irw ems
cr eal talrl fmnmdia olvinbt fdro i mugslde pnsga?

Vssu o iet nal airu y mjavn teiw peei mssr o uk
rvfn rv pp sr te saes st
dmr ogn niih axnu eect aeb a eppl tee iop efdd.

Kerc ahie di dmer ft ltb cre tnt
sunt thidxmler sen rasdedon ntnsroi tpnmm y sni ocagu
ayvoanrtl i eauysn esstnue rtoin pscfm afuarov tnsxcuiom lits tn
rishnig xliylef sitd oo jeut sorst crtrag io rxva
ra mr odgtalo y abspot y tdtu enseiad of tfh to eiyo?

Heafhei gmb uexennr ursrhea tds srna besiant mvmc sedihn htr
teayr beauaianu inrehechx adf ree trieetse hntts cooads ne
evl nooe ipmuor rnrg etaz ceex ltrv itant fet eiaus
in rt nt tieus pg nae nfan ihfe
wnsn eteea oire etasu trc aco errn
nvnl nmc yeeh eio ydi apoy ees hxgs ecnn
obx fte dis fro yctd thfr o dntt iac enee drr
ptrsi edciw ynt raeeya ioednht a edas hcc
tir cpoa eni nv neee da tas deds imcu whoe!

Dnngh esda itnd i daou pei fr o yu
hasewn aeac ynpvhbm iprsri fftr gsu rvrot ro
ekim tsca guterf tnnar otdee ioihdd erilev fnrm es?

Vilirad aasalnao ue eefckifxo rhefad ew ybrvao hlros
er smdtzln aiohy y war rweq uc witeoas ncrts
tsindr lget rtcpd a ftsc tyi o aig nr?

Neeib tflll iea anree aht een fsrz oar oeman
nftkm th rhyoee tlndsou paoef a onhr eel ueir aefus.

Ipoer rfhd oeyd eidd tsgt pen tvrth
in ceiapa rrs ngih hrse etitu fmrgacf dpee
ssid yufte netc asdmd rai y tw rrt oper eat srod
otoe crp o baeaton iy to ecaosdb dyc
oesnd tga ctns llurt igaer dawr qh ensp im egr
rcare oster nteot tetidg nehlet cnrn ageisaf rrr lr
ar eerru a ssacff ted eldnsei sssmenl nwae cy hetm?

Vcua itr raut wro hdcf eetm apt nte edwr eo
rlner eehgeh o lsti eii iit illeeetg y ntod
nnttt ihne fcd illen woi veoea ye?

Esbec inen tmhnre tvitea smcdtfp any eaate!

Zndy etsetl aoyysde eee mrr iiasti qndnvdeh envm.

A ege mirdu arqgj nraie iotmr dts tdrgh ani
weto ier etbe sear brl sce hte xrh coe
vineyg aryodtg dil prso cer hsth iweyr?

Y ctr rape umn iro eoa pad riine
rmanhttt oerelmo ooiaenit rdrenyi arf nnaefdot unccetoq caara a atede jia?


author: Joe Greco

subject: Re: Musings upon the concept of creating reliable news services with redundant sets of unreliable servers....

text:
In news.software.nntp article <[email]>, [email] (Brad Knowles) wrote:
:Folks,
:
:    I've been thinking about this problem quite a bit within the last few
:days.  The goal is to avoid Single Points of Failure (SPOFs), and what
:I've had rattling around in the back of my brain is something along the
:lines of RAIS (Redundant Arrays of Inexpensive Servers [0]).
:
:
:    It's easy to set up multiple news peering servers, and have each one
:handle 1/n of your news peers, and you can use "virtual" path elements
:that you tell your peers to look at (when considering feeding you back an
:article), but internally you instead pay attention to the host path
:element that gets the virtual path element appended on each server.
:
:    I think I understand this part, but of course would like to hear any
:thoughts that others might have.
:
:    You can set up a "master" news spool server and one or more "slave"
:news spool servers that get their article numbers, etc... from the master,
:and should the master die you can relatively quickly turn one of the
:slaves into a replacement master, etc....
:
:    Again, I think I understand this part.
:
:    However, now comes the problem of setting up multiple redundant news
:reader servers.  If you have them NFS mount the news spool, obviously they
:can only mount that from one server.  This creates a SPOF.
:
:    Based on previous bad experiences, I wouldn't bother looking at
:distributed file systems (which might implement multiple redundant copies
:of files) such as AFS, DFS, or Coda, although perhaps I'm wrong in this
:matter.
:
:    I'm leaning towards an NNTPCache-type solution, but I'm wondering how
:you would tell it to pull the same set of newsgroups from more than one
:server.  I'm also wondering how you would ensure that the article
:numbering, expiration, newsgroup file, etc... would stay in sync across
:multiple NNTPCache servers.  And how would you then extend this concept to
:allow for off-site "backup" news servers, perhaps run by a news
:outsourcing operation?
:
:    I think this part is where I could use the most help.  Any thoughts?
:
:[0]  In this case, "Inexpensive" is a relative term.  One news spool
:server built on top of high-end PC server hardware and running a
:relatively robust freely available OS (I would choose FreeBSD, but that's
:a personal preference) can be *quite* a bit less expensive (three to five
:times less expensive) than a well-configured DECompaq Alpha that has been
:configured throughout for internal redundancy and fault resiliance, and I
:have to assume that the same is true for Sun Solaris (the only other
:option I'd be likely to consider).
:
:In this case, I'd be willing to spend less money to buy multiple servers,
:and get my redundancy through having more machines than in having more
:expensive but more capable single machines.

The cost of the machine is not an issue.  A good PC server class system
can be built for ~$2K-$3K, but it is the I/O subsystem, disks and RAID
controller, that will co$t you to death.

Using NFS is about the worst possible thing I can think of.  NFS is very
cruddy, unreliable, and noisy - causing system hangs in the event of
problems, etc.

You are going in the right direction, you just don't have the right
technology.  You want big spool servers.  Machines that have lots of
disk and lots of speed but aren't being bogged down by dealing with
the inane stuff clients like to spew at a NNRP daemon.  You want
smaller reader servers.  Machines with small amounts of disk that
handle the NNRP transactions.  Machines that handle managing the
overview data, active files, etc.

You want to replace NFS with NNxP.

Look at Diablo.  Matt Dillon has done a fantastic job of solving your
problem for you.  You set up reader boxes running dreaderd.  These
accept a header-only feed from a master server, and create an overview
database from the Xref: data and other headers.  This is very low-cost
in terms of bandwidth, etc.  Each reader can handle hundreds of users,
and talks to the spool servers via a dozen or so NNTP channels which
are used for "ARTICLE <mess@geid>" type retrievals - very easy and very
fast for the spool server to do.

You do not have to have one spool server that has all your articles.
You can have a spool server for binaries (with big disks) and one for
regular articles.  You can have redundant spool servers.  You can do
lots of neat stuff from an architectural point of view.  Can you say
"implement a geographically/topologically diverse spool architecture
and put reader machines at every major POP"?

I'm heavily committed to this paradigm at this point.  It is conceptually
correct.  Your back end spool servers can be anything that is able to
answer "ARTICLE <mess@ageid>"...  INN, Typhoon, Diablo.  Your front
end reader servers are small and cheap to build.

I'm currently shooting to build such a system that will scale into the
millions-of-subscribers range.  I've solved a number of scalability
issues, implemented some fairly comprehensive audit trail facilities in
Diablo, and done a whole bunch of other neat and cool stuff.  It works.
This article is being posted through one of my test servers.

Yes, you still have a SPOF... the master server.  I know that I don't
care about this.  It is a small SPOF that can be handled by keeping a
hot spare machine around, and it need not be a huge system since it
only really needs to keep a days worth of news or so.  When it fails,
you pull the dactive from a reader and away you go.  Maybe you lose
some articles in the process if you're sloppy.

Now everybody knows why I decided to dump my INN development efforts...
:-)

... JG


author: Brad Knowles

text:
In article <378cc595$0$[email]>, [email] (Joe Greco)
wrote:

    Yes, of course.  And a Mylex DAC960SXI controller with five channels,
256MB cache, battery-backup, etc... will cost about 3x that of the machine
it's connected to (yes, I've done the math ;-).  And 9GB 10kRPM disks from
either Seagate or IBM will cost around $600/each, at least from the best
prices I've seen.

    However, these controller costs and disk media costs are *greatly*
exaggerated in the DECompaq/Sun solution, so not only do you end up paying
more for the somewhat more reliable machine, you also end up paying a lot
more for the priviledge of buying your I/O subsystem from them.

    I figured that part.

    I also grokked that.  I could easily re-use existing Pentium Pro 200
servers, so long as they've got decent OSes and disks underneath them to
do the job.

    If I use something like NNTPCache, I'll want to replace the disks with
small striped arrays of fast disks.  If I go with something like
Diablo/dreaderd, then local disk writes should be virtually non-existant
(except for keeping track of the history file, logging, etc...), and
single replacement reliable and high-speed drives should be sufficient.

    NFS was never really a player in the new model I've been trying to build.

    Hrm.  So the news flows into the master server, which sends a
header-only feed to the reader servers, but then how do the articles get
written with the same article ids, etc... on the news spool servers?
Wouldn't that also require a header-only feed, but then wouldn't you also
need to follow that up with a full article feed?

    I'm confused regarding how articles flow from the master server to the
news spool servers.  I think I understood the rest.

    My plan was actually to be redundant for !alt.binaries.*, and to split
alt.binaries.* across the servers (so that I don't waste all that disk
space across each and every news spool server).

    That's kind of where I'm headed.  I'm just not yet quite convinced
that I need to use Diablo to get there (although you're doing a pretty
good job of convincing me ;-).

    MOOF!!!

    I was thinking pretty much the same thing.  Some SPOFs aren't
avoidable (there has to be either just one machine or some sort of locking
mechanism used across multiple machines in order to guarantee unique
sequential article numbers), but at least you can work to avoid the rest
and make the machine that is the SPOF as reliable and easy to quickly
replace as possible.

    I've been wondering what the heck you've been up to....  ;-)


author: bill davidsen

text:
In article <378cc595$0$[email]>,
Joe Greco <[email]> wrote:

| The cost of the machine is not an issue.  A good PC server class system
| can be built for ~$2K-$3K, but it is the I/O subsystem, disks and RAID
| controller, that will co$t you to death.

That somewhat depends on what you want to do with the machine. If you
use software RAID in the o/s you can keep the cost low, and for the
redundant reader machines you only need raid-0 for performance. The i/o
is always the most expensive part of a large system, even back in the
days of memory at $4Million/MB mainframes (1960's).

| Using NFS is about the worst possible thing I can think of.  NFS is very
| cruddy, unreliable, and noisy - causing system hangs in the event of
| problems, etc.

I ran a bunch of NFS slaves in 1.4 and 1.5, and they had none of those
problems, they were just slow and used a lot of bandwidth. Had no hangs
or bad data, just poor performance. We agree that's this is not the way
to go, for whatever reasons.

| You are going in the right direction, you just don't have the right
| technology.  You want big spool servers.  Machines that have lots of
| disk and lots of speed but aren't being bogged down by dealing with
| the inane stuff clients like to spew at a NNRP daemon.  You want
| smaller reader servers.  Machines with small amounts of disk that
| handle the NNRP transactions.  Machines that handle managing the
| overview data, active files, etc.
|
| You want to replace NFS with NNxP.
|
| Look at Diablo.  Matt Dillon has done a fantastic job of solving your
| problem for you.  You set up reader boxes running dreaderd.  These
| accept a header-only feed from a master server, and create an overview
| database from the Xref: data and other headers.  This is very low-cost
| in terms of bandwidth, etc.  Each reader can handle hundreds of users,
| and talks to the spool servers via a dozen or so NNTP channels which
| are used for "ARTICLE <mess@geid>" type retrievals - very easy and very
| fast for the spool server to do.

One of the problems with having an anything feed is that it complicates
the job of getting a new reader machine online. That's one of the few
good features of NFS, it allows a new rader box to go from unpack to in
service in a few hours. Any form of a header feed requires that the
master be able to generate such a feed as a catchup.

| Yes, you still have a SPOF... the master server.  I know that I don't
| care about this.  It is a small SPOF that can be handled by keeping a
| hot spare machine around, and it need not be a huge system since it
| only really needs to keep a days worth of news or so.  When it fails,
| you pull the dactive from a reader and away you go.  Maybe you lose
| some articles in the process if you're sloppy.

I'm still not sure how you keep the article numbers in a newsgroup
machine in sync on each reader. It's desirable to be able to move
readers from one to another of these machines, so you sort of have to
make them appear to be the same. Browsers are fragile, users are easily
upset, I like to address the technical problem of not having them see
changes rather than the political problem of dealing with unhappy
users.

The ideal user population is a steady state load used to help me
evaluate my clever technical solutions, not an evaluation committee.


text:
[email] (Joe Greco) wrote:

I've been thinking about that type of design for years.  It's always
sounded very appealing to me.  Creating a network of cheap servers
is easy to do where as trying to create one large server with the same
power is far more expensive.

It's why I started writing my own news server back in December.  I wanted
to create an archetecture which would allow the server do be distributed
over a redundant array of cheap servers.

This is just what I've done with my new server.  I just put
it into production on NewsReader.Com two days ago replacing my old
inn server.

I haven't really looked at what Matt's done so please excuse me for being
stupid about this...

So each reader box maintains it's own overview database which includes the
indexes required to map group/article# into message-ID?  How do they know
when an article has expired from a spool server?

Is there some regular overview expire process which scans everything on a
regular basis?  Or do the readers just expire their data based on their own
logic (which might or might not match the back-end servers), or what?

What I did was to have each spool server number the articles sequentially.
And then, instead of requesting articles by message-id, you get them by
their number.  The storage scheem the servers use are set up to make this
simple and fast for them to do.

The numbers are designed to wrap at a fixed point so there's no problems
with the numbers getting to large.  It only limits how many articles one
server can store.

And, articles always expire from the spool servers in order.  So this
means, that at any time, each spool has a range of current "valid" article
numbers. The front-end server (which currently is also the master server)
always knows what the current valid range is for all the spool servers -
and it uses that to quickly know if any article is still valid, or if it
has expired.

There's a group/article# to spool/spool-article# database maintained by
the master server (though this could be dupped in all the front-end servers
like Diablo).  So, to check to see if article #12345 in group x.y.z has
been exired, all the server has to do is look up the spool, and article
number on that spool, and then check that against it's table which is
constantly being updated.

This allows the front end overview information to stay perfectly in sync
with the back end spool servers.  This way, you never report "old" overview
info which could lead to extra "article no longer available" messages for
the users.

It also comes into play in the history file managment.  Knowing if an
article is still valid allows you to create a history file that starts
small, and only expand if needed. If it's expensive to figure out if an
article is still valid, then you have to use a fixed size history file and
force the admikn to make sure it's big enough (like Typhoon), or expire
later instead of in real time (like old Diablo at least).

I've even been looking a bit at low-end consumer PCs to use as the spool
servers.  They are so cheap these days that you can get an entire PC with
large IDE drives for less money than a bare SCSI drive of the same size.
I'm not sure how well they would actually perform, but it's an interesting
idea made possible by this type of architecture.  And since ethernet is hot
swapable, it's easy to replace these things on the fly (assuming your
software is set up to deal with that).

Yeah, I too am heading towards that.

I have the structure in place to do a lot of this, but currently
the master server is doing all the reading as well, so it doesn't
yet scale well - still needs more code....

And most likely you will have some networking hardware which is also
a SPOF.  But if the reliability is high enough, this is fine.

And the later it is said:

Interesting twist.  In my design, the master gets a full feed, assigns the
article numbers (and updates the Xref header in the article), then sends
the article to a spool.  The spool assigns it's spool-number to the
article, and returns that info to the master.  Only then (after the master
knows for sure the spool correctly saved it) does the master update the
overview-like database and make the article "visible" to NNTP clients.  My
master also keeps two high water mark values for each newsgroup.  One is
used for assigning new numbers, but the second is only updated to reflect
when articles are stored and made visible to the NNTP clients.  The NNTP
clients only see the high-water mark that correctly matches what is
currently available for them to read.

What do you do about the Xref: line?  When you read an article does the
reader box insert it on read?  Or do you just not get Xref: lines from your
system?


text:
In news.software.nntp article <[email]>, [email] (Brad Knowles) wrote:
:In article <378cc595$0$[email]>, [email] (Joe Greco)
:wrote:
:> The cost of the machine is not an issue.  A good PC server class system
:> can be built for ~$2K-$3K, but it is the I/O subsystem, disks and RAID
:> controller, that will co$t you to death.
:
:    Yes, of course.  And a Mylex DAC960SXI controller with five channels,
:256MB cache, battery-backup, etc... will cost about 3x that of the machine
:it's connected to (yes, I've done the math ;-).  And 9GB 10kRPM disks from
:either Seagate or IBM will cost around $600/each, at least from the best
:prices I've seen.

That's because you costed it with all that RAM.  A DAC960SXI with 5
channels, 16MB RAM, etc., will cost about 20% more than the machine it
is attached to...  assuming the type of machines I've been spec'ing in
recent months.  (ASUS P2B-DS ~= $425, case ~= $250, 2 x PII-xxx ~= $400,
1GB RAM ~= $1100 ...  DAC960SXI ~= $2600)  I've been recycling 64MB SIMM
modules and putting 128MB in my DAC's, and since I don't have a reasonably
priced source for 128MB SIMM's, I'm happy with that.

:    However, these controller costs and disk media costs are *greatly*
:exaggerated in the DECompaq/Sun solution, so not only do you end up paying
:more for the somewhat more reliable machine, you also end up paying a lot
:more for the priviledge of buying your I/O subsystem from them.

Rich folks, fools, and idiots buy storage systems from hardware vendors.

:> You are going in the right direction, you just don't have the right
:> technology.  You want big spool servers.  Machines that have lots of
:> disk and lots of speed but aren't being bogged down by dealing with
:> the inane stuff clients like to spew at a NNRP daemon.
:
:    I figured that part.
:
:>                                                         You want
:> smaller reader servers.  Machines with small amounts of disk that
:> handle the NNRP transactions.  Machines that handle managing the
:> overview data, active files, etc.
:
:    I also grokked that.  I could easily re-use existing Pentium Pro 200
:servers, so long as they've got decent OSes and disks underneath them to
:do the job.
:
:    If I use something like NNTPCache, I'll want to replace the disks with
:small striped arrays of fast disks.  If I go with something like
:Diablo/dreaderd, then local disk writes should be virtually non-existant
:(except for keeping track of the history file, logging, etc...), and
:single replacement reliable and high-speed drives should be sufficient.

I'm not confident of that, I'm still speccing my reader machines to be
quad drive machines.  The cost is still below the price dictated by the
design.

:> You want to replace NFS with NNxP.
:
:    NFS was never really a player in the new model I've been trying to build.

Yes, but you said the evil letters anyways.

:> Look at Diablo.  Matt Dillon has done a fantastic job of solving your
:> problem for you.  You set up reader boxes running dreaderd.  These
:> accept a header-only feed from a master server, and create an overview
:> database from the Xref: data and other headers.  This is very low-cost
:> in terms of bandwidth, etc.  Each reader can handle hundreds of users,
:> and talks to the spool servers via a dozen or so NNTP channels which
:> are used for "ARTICLE <mess@geid>" type retrievals - very easy and very
:> fast for the spool server to do.
:
:    Hrm.  So the news flows into the master server, which sends a
:header-only feed to the reader servers, but then how do the articles get
:written with the same article ids, etc... on the news spool servers?
:Wouldn't that also require a header-only feed, but then wouldn't you also
:need to follow that up with a full article feed?

No.  You missed the point.  The readers and the master are the only places
where there is knowledge of so-called "article numbering".  The readers
then have access to the Message-ID:, as part of the overview, and the
spool servers don't need to be able to handle things like GROUP commands
and the heavy duty indexes that are associated with them.  Doing an
"ARTICLE <mess@geid>" type command is very lightweight:  maybe a history
access (if it isn't in RAM already) and then an article retrieval.

:    I'm confused regarding how articles flow from the master server to the
:news spool servers.  I think I understood the rest.

Uh, actually, im my implementation - they _don't_.  Articles flow from the
news spool servers (where article numbering is irrelevant and useless) in
_to_ the master server.  This is simply to assure that an article is there
on at least one spool server before the master assigns it a number and
passes that on to the reader machines.  The spool servers are directly
peered with my transit servers, keep no active file, and have only one
outgoing feed - to the master.  They just suck in the data sent by the
transit servers, store it, and feed it to the master.  The master assigns
article numbers, passes that on to the readers via a header-only feed.
Each reader builds its independent overview and active file databases, and
when a user goes into "alt.sex" and asks for "ARTICLE 1", the reader gets
the overview data for alt.sex:1, parses out the Message-ID:, and then
goes around asking each spool server for that Message-ID.

:> You do not have to have one spool server that has all your articles.
:> You can have a spool server for binaries (with big disks) and one for
:> regular articles.  You can have redundant spool servers.
:
:    My plan was actually to be redundant for !alt.binaries.*, and to split
:alt.binaries.* across the servers (so that I don't waste all that disk
:space across each and every news spool server).

You lose then in the event that a server is down (etc).  Diablo handles
it fairly well, though, so if that is acceptable to you, so be it.

:>                                                           You can do
:> lots of neat stuff from an architectural point of view.  Can you say
:> "implement a geographically/topologically diverse spool architecture
:> and put reader machines at every major POP"?
:
:    That's kind of where I'm headed.  I'm just not yet quite convinced
:that I need to use Diablo to get there (although you're doing a pretty
:good job of convincing me ;-).

Write to me in e-mail if you need more convincing.

:> I'm currently shooting to build such a system that will scale into the
:> millions-of-subscribers range.  I've solved a number of scalability
:> issues, implemented some fairly comprehensive audit trail facilities in
:> Diablo, and done a whole bunch of other neat and cool stuff.  It works.
:
:    MOOF!!!

Uh, what was that...  I hope you're not laughing at me.  :-)

:> Yes, you still have a SPOF... the master server.  I know that I don't
:> care about this.  It is a small SPOF that can be handled by keeping a
:> hot spare machine around, and it need not be a huge system since it
:> only really needs to keep a days worth of news or so.  When it fails,
:> you pull the dactive from a reader and away you go.  Maybe you lose
:> some articles in the process if you're sloppy.
:
:    I was thinking pretty much the same thing.  Some SPOFs aren't
:avoidable (there has to be either just one machine or some sort of locking
:mechanism used across multiple machines in order to guarantee unique
:sequential article numbers), but at least you can work to avoid the rest
:and make the machine that is the SPOF as reliable and easy to quickly
:replace as possible.

This is both a strength and a weakness of Diablo.  The structure of the
system allows a single Message-ID to be assigned several times into a
group, since the reader does not maintain its own history.  This does
allow you to feed the same article with the same Xref: data in several
times, without problems, but in the event where you were to lose the
history on your master, you could end up with duplicate stuff in your
overview.

I do not think that is a bad thing.  :-)

:> Now everybody knows why I decided to dump my INN development efforts...
:> :-)
:
:    I've been wondering what the heck you've been up to....  ;-)

Lots of trouble.

... JG


text:
In news.software.nntp article <19990714184455.693$[email]>, [email] (Curt Welch) wrote:
:[email] (Joe Greco) wrote:
:> In news.software.nntp article
:> <[email]>, [email]
:> (Brad Knowles) wrote:
:
:> :    I've been thinking about this problem quite a bit within the last
:> :few days.  The goal is to avoid Single Points of Failure (SPOFs), and
:> :what I've had rattling around in the back of my brain is something along
:> :the lines of RAIS (Redundant Arrays of Inexpensive Servers [0]).
:
:I've been thinking about that type of design for years.  It's always
:sounded very appealing to me.  Creating a network of cheap servers
:is easy to do where as trying to create one large server with the same
:power is far more expensive.
:
:It's why I started writing my own news server back in December.  I wanted
:to create an archetecture which would allow the server do be distributed
:over a redundant array of cheap servers.
:
:> You are going in the right direction, you just don't have the right
:> technology.  You want big spool servers.  Machines that have lots of
:> disk and lots of speed but aren't being bogged down by dealing with
:> the inane stuff clients like to spew at a NNRP daemon.  You want
:> smaller reader servers.  Machines with small amounts of disk that
:> handle the NNRP transactions.  Machines that handle managing the
:> overview data, active files, etc.
:>
:> You want to replace NFS with NNxP.
:
:This is just what I've done with my new server.  I just put
:it into production on NewsReader.Com two days ago replacing my old
:inn server.
:
:> Look at Diablo.  Matt Dillon has done a fantastic job of solving your
:> problem for you.  You set up reader boxes running dreaderd.  These
:> accept a header-only feed from a master server, and create an overview
:> database from the Xref: data and other headers.  This is very low-cost
:> in terms of bandwidth, etc.  Each reader can handle hundreds of users,
:> and talks to the spool servers via a dozen or so NNTP channels which
:> are used for "ARTICLE <mess@geid>" type retrievals - very easy and very
:> fast for the spool server to do.
:
:I haven't really looked at what Matt's done so please excuse me for being
:stupid about this...
:
:So each reader box maintains it's own overview database which includes the
:indexes required to map group/article# into message-ID?

That's actually _all_ the overview databases are...  headers from the
actual Usenet articles, stored in an index to map group/article# to
header.  That information naturally includes the MID.

:How do they know
:when an article has expired from a spool server?

They don't.  It is actually an extremely elegant design, because it allows
so much flexibility.  You can have a small local spool under your admin
control, and then also have a slower "backup" link to someplace like
newsreader.com or remarQ.com which retrieves articles not found on the
local spool.

Realistically, there is no good general way to keep the overview and
spools completely sync'ed and yet maintain the ability and flexibility to
keep them on separate machines or even separate administrative realms.

You can design more complex mechanisms that greatly reduce your
flexibility in what clever things you can do...  but this adds great
quantities of complexity.  What if you have lots of "small" spools,
each of which hold the last week of news, but you choose to have one
"retention" spool in your company that holds the last month of news?
Even assuming you had some hypothetical package that allowed the
overview database to know when the articles had been removed from
the spool, the overview database would still have to be clever enough
to wait for the last instance of the article to vanish from the spool
servers before removing the overview data.  What about a heterogeneous
environment where one of the spool servers isn't even the same software
package?  Right now, my new Diablo servers rely somewhat on the current
INN reader system to provide some types of articles for which I have
not yet built Diablo spool servers.

:Is there some regular overview expire process which scans everything on a
:regular basis?  Or do the readers just expire their data based on their own
:logic (which might or might not match the back-end servers), or what?

Based on their own logic... which is theoretically based on what the
admin knows about the actual environment.  Querying each of potentially
dozens of spool servers to make sure that an article no longer exists
on any of them would be bad for a single reader, and disastrous for a
large number of readers.  You could do it, sure, but I don't think you
would _want_ to.

:What I did was to have each spool server number the articles sequentially.
:And then, instead of requesting articles by message-id, you get them by
:their number.  The storage scheem the servers use are set up to make this
:simple and fast for them to do.

Diablo makes it fast and easy for them to request by MID.  Different
way to implement the same general thing.  However, if you mean that you
are storing them by their group.name:number, then you are pushing extra
work up to your spool server because it is having to maintain those
indexes in some fashion or another.  The Diablo spool box just lays the
articles down and writes history entries, and retrieval is equally
simple because the history lookup on the MID gives you the reference
to the actual article, which is then trivial to fetch and send to the
client.  It sounds like you have to go through some more complex
data to get to the actual message, although if it was coded very
efficiently, I guess it wouldn't be an issue.

:The numbers are designed to wrap at a fixed point so there's no problems
:with the numbers getting to large.  It only limits how many articles one
:server can store.

Same with Diablo, except that it is simply doing its normal expiration
thing and storing by MID.

:And, articles always expire from the spool servers in order.  So this
:means, that at any time, each spool has a range of current "valid" article
:numbers. The front-end server (which currently is also the master server)
:always knows what the current valid range is for all the spool servers -
:and it uses that to quickly know if any article is still valid, or if it
:has expired.

That sounds like more dependence than I'd personally like to see.  In my
environment, I don't care too much if I newfs my master, the only real
damage I might cause is due to the loss of history, so I might get some
duplicate messages showing up on the readers.  Whoopie.

:There's a group/article# to spool/spool-article# database maintained by
:the master server (though this could be dupped in all the front-end servers
:like Diablo).  So, to check to see if article #12345 in group x.y.z has
:been exired, all the server has to do is look up the spool, and article
:number on that spool, and then check that against it's table which is
:constantly being updated.

Oh, wait, your spool/spool-article# is not the same as group/article#?
Ich.  I see what you're trying to do, but I don't think I like it too
much.  It lacks the simple elegance of Diablo, although it may allow
you a particular form of flexibility in how you choose your retention
policy.

:This allows the front end overview information to stay perfectly in sync
:with the back end spool servers.  This way, you never report "old" overview
:info which could lead to extra "article no longer available" messages for
:the users.

I just make sure I keep the data on the spool servers longer than the
readers store the overview data.  I'll concede that that is tricky and
a bit speculative.

:It also comes into play in the history file managment.  Knowing if an
:article is still valid allows you to create a history file that starts
:small, and only expand if needed. If it's expensive to figure out if an
:article is still valid, then you have to use a fixed size history file and
:force the admikn to make sure it's big enough (like Typhoon), or expire
:later instead of in real time (like old Diablo at least).

Diablo still does batch expiration.  It has not shown itself to be any
sort of bottleneck in my experience.  The Diablo reader stuff uses a
dynamically sizing (and it uses some method I don't like, I plan to
look at this soon and change it) overview index on a per-group basis.

:> I'm heavily committed to this paradigm at this point.  It is conceptually
:> correct.  Your back end spool servers can be anything that is able to
:> answer "ARTICLE <mess@ageid>"...  INN, Typhoon, Diablo.  Your front
:> end reader servers are small and cheap to build.
:
:I've even been looking a bit at low-end consumer PCs to use as the spool
:servers.  They are so cheap these days that you can get an entire PC with
:large IDE drives for less money than a bare SCSI drive of the same size.

Yeah, I considered that for some other purposes...  using a Promise (I
think) IDE controller in addition to two on-boards, one can get four
IDE channels and eight IDE drives on a single system, which reduces the
cost overhead of the PC somewhat.  I still like SCSI better, since I can
put hundreds of drives on a single machine, and I have a lot more faith
in SCSI to handle server-class loads.

:I'm not sure how well they would actually perform, but it's an interesting
:idea made possible by this type of architecture.  And since ethernet is hot
:swapable, it's easy to replace these things on the fly (assuming your
:software is set up to deal with that).

It should be!

:> I'm currently shooting to build such a system that will scale into the
:> millions-of-subscribers range.
:
:Yeah, I too am heading towards that.
:
:> I've solved a number of scalability
:> issues,
:
:I have the structure in place to do a lot of this, but currently
:the master server is doing all the reading as well, so it doesn't
:yet scale well - still needs more code....
:
:> Yes, you still have a SPOF... the master server.
:
:And most likely you will have some networking hardware which is also
:a SPOF.  But if the reliability is high enough, this is fine.

With dual Ethernets and other arrangements as well, I can eliminate the
networking hardware as a SPOF.  One requirement in my current design is
that everything have at least two paths to each primary spool server.

:And the later it is said:
:> :    I'm confused regarding how articles flow from the master server to
:> :the news spool servers.  I think I understood the rest.
:>
:> Uh, actually, im my implementation - they _don't_.  Articles flow from
:> the news spool servers (where article numbering is irrelevant and
:> useless) in _to_ the master server.  This is simply to assure that an
:> article is there on at least one spool server before the master assigns
:> it a number and passes that on to the reader machines.
:
:Interesting twist.  In my design, the master gets a full feed, assigns the
:article numbers (and updates the Xref header in the article), then sends
:the article to a spool.  The spool assigns it's spool-number to the
:article, and returns that info to the master.  Only then (after the master
:knows for sure the spool correctly saved it) does the master update the
:overview-like database and make the article "visible" to NNTP clients.  My
:master also keeps two high water mark values for each newsgroup.  One is
:used for assigning new numbers, but the second is only updated to reflect
:when articles are stored and made visible to the NNTP clients.  The NNTP
:clients only see the high-water mark that correctly matches what is
:currently available for them to read.

That's nice, but almost required for your stated design.  I get away with
only having to worry about making sure that it is available on _some_
spool server in the pool, and by arranging the feeds such that the master
only gets articles from the spool servers, that is actually designed right
in and guaranteed by the setup.  Other than that, Diablo is doing the
same basic thing with a lot less chattiness...  when the master hands the
reader a header, it becomes "available for reading".  There is a potential
window where it is available on A but not B, but that will happen to yours
too as you scale it, I believe.

:What do you do about the Xref: line?  When you read an article does the
:reader box insert it on read?  Or do you just not get Xref: lines from your
:system?

This is one odd thing.  But it is a side-effect of policy decisions that
I have made, it is not really a Diablo restriction.

When you ask for "ARTICLE 1", Diablo provides the header from the overview
(which of course has an Xref: for sync purposes) but searches the spools
for the body of the associated MID in the overview.

When you ask for "ARTICLE <x@y>", Diablo searches the spool servers
directly for "<x@y>", and returns the complete article.  This makes sense
since the reader code does not maintain a history, and cannot map that
request to an overview entry.

In my particular design, then, no, you don't get an Xref: line.  If that
bothers you, you can design it such that the spool servers are fed by the
master and do get Xref: lines, but then you lose the automatic availability
guarantee that we just discussed, and you have to make sure that your spool
servers will have the article.

You could do this by making the master Xref: the articles and feed the
spools, then have the spools provide the header-only feeds to each of the
readers.  But, then in my case, you are hauling more data than necessary
around across your network - which is why I decided against doing that.

Or you could fudge it and hope for the best.

Either way, I like the "lego effect" Matt has built into the overall
design of the system, you can do lots of neat and cool things.

... JG


text:
In article <378cf321$0$[email]>, [email] (Joe Greco)
wrote:

    I think you're using US prices.  Unfortunately, although we are
registered as a reseller with Ingram Micro, we still have to pay heavy
duties and import fees for stuff we don't buy locally, so prices over here
in Belgium are greatly inflated, although they're not necessarily all
inflated to the same degree.

    In fact, we've found that we can buy systems from a certain name brand
vendor at a price cheaper than we can buy no-name systems that are kludged
together in someone's basement, and even cheaper than what we would pay
for them if we directly bought all the pieces ourselves and put them
together.

    However, on the other side, the DAC960SXI only goes up in price when
it comes over here, no matter how it arrives.  So, it gets relatively
quite a bit more expensive than the machine it's going to be connected to,
and that machine isn't exactly a low-end configuration.  We're even
ponying up for ECC RAM, which isn't cheap.

    Better to be safe than sorry!  I was planning on having the reader
machines be quad drive to begin with, and I've never really waivered from
that plan.  I'm glad that I've now got firm confirmation that this is
probably a good idea.

    Hmm, so the same article flows in separately to the spool servers,
which might issue a different local article number for the same
message-id, and then to the master server?

    Interesting.  Very, very interesting.

    Does the header-only feed from the master to the readers also include
the article number, so that all reader servers have exactly the same
picture of what article number maps into what message-id, which they can
then query for from the news server regardless of how the news server
issues it's own article numbers or how it stores the article?

    Also, what happens with cross-posts and Xref headers, where article
numbers are used and not message-ids?

    And how do you get a news reader server to use multiple spool servers,
either load balancing or preferring one over the other(s)?

    It seems to me that there's a lot to Diablo/dreaderd that I don't
understand.

    Yeah, I know.  I just don't want to waste hundreds of GB of disk space
duplicating binary content that's going to turn over in a few days
anyway.  If there's a catastrophic crash and loss of a news server that
toasts some portion of the spool for binary articles, they'll start
filling up again soon enough.

    Will do.  Thanks for the offer, although you may come to regret it if
I pester you too much.  ;-)

    No, that was not laughing at you.  That was my ASCII equivalent of
snorting coffee [-1] out my nose as I read your description and was rather
impressed by the scale of your plans. [0] [3] [4]

    Hmm.  I'm not sure I quite understand this point.  Could I get you to
explain in a bit more detail?

        -Brad

[-1] No, I wasn't actually drinking coffee at the time, but I'm not about
to say that I was snorting Coke out of my nose, not even Diet Coke or Coca
Light as they call it over here.  ;-)

[0] Although, technically, I was quoting Clarus the dogcow. [1] [2]

[1] The virtual mascot of the short-lived Claris corporation that was spun
off and then re-absorbed into Apple.

[2] If you've ever done a "Page Setup..." on a Macintosh, you've seen
Clarus used as the icon to indicate whether the paper is in landscape or
portrait mode, etc....

[3] Sigh....  It's been too long since I left AOL.  I'm starting to think
small again....

[4] Ack!!!  Rampant ASR-like footnote-itis!  ;-)


author: Simon Lyall

text:
Joe Greco <[email]> wrote:

Alternatively you could have a diablo server on your reader box handling
multiple incoming header feeds to weed out dups. It could get a bit ugly
running article, header and reader servers all on the same machine (for
smaller pops) but I can't see any obvious problem with it.

A possible idea would be to have a backup main server that normally runs
in Xref slave mode. When your normal one bursts into flames the backup
stops and restarts itself in Xref-creating mode. I'll have to check
exactly how I was going to do this though.