NEWSX

Section: Maintenance Commands (8)
Index Return to Main Contents

NAME

newsx - news server exchange; post and fetch news articles

SYNOPSIS

newsx [ options ] [ spoolname [ [ hostname ] port ] ]

DESCRIPTION

newsx is a client connecting to an NNTP server, posting outgoing articles batched by the news system, while also fetching new incoming articles from the same newsserver.

It provides the NNTP capabilities required for serving a small local news spool such as those that may be found on installations with NNTP access through limited ISP accounts only, as well as being suitable for exchanging news with supplemental news sources for full scale news servers.

OPERATION

The program would normally be run by user news.

When started, newsx will go through the spoolname outgoing spool queue, and attempt to post all articles therein to the NNTP server hostname.

If a username and password for use with the simple authorization protocol have been supplied, they will be sent to the server first. If the -r option is specified, a MODE READER command will be sent.

When the outgoing transfer is finished, articles will be removed from the outgoing spool queue if successful transfer has been acknowledged by the NNTP host, if the article was specified by the host as a duplicate (already posted), or if a spooled article can no longer be found.

The fetch phase is then initiated. Articles available from the newsserver will be fetched and placed in the incoming spool, ready to be injected into the news system by newsrun (C News) or rnews -U (INN).

The groups are visited in the sequence that they appear in the host incoming state file, see in.hosts(5). Group names not present in the active file will not be fetched.

Every group name is checked against the spoolname entries in the newsfeeds (INN) or sys (C News) file, as well as any --groups option, and only groups allowed there will actually be fetched. For C News, the group name is also checked against the ME entry, whereas in INN the ME entry is just prepended to the spoolname entry.

In this way, the newsfeeds/sys file is used to control which newsgroups are actually received from the remote site, in the same manner that the local server determines which outgoing article will go where.

If an -e option hasn't been specified, the local active file is then traversed to see if there are groups not mentioned in the current host newsrc file. Again, only groups allowed by newsfeeds or sys as well as any --groups are actually fetched. Since these groups don't have a starting point specified, all available articles will be fetched unless the --maxnew option dictates otherwise.

To avoid transferring cross-posted articles more than once, articles with messags IDs already fetched in the same session will not be fetched again.

To allow multiple news sources to be utilized in an efficient manner, articles present in the history database will be skipped.

The pull phase can be omitted entirely by using the -g option.

Statistics on the connection will be logged via syslog(2) (or stderr with -d). The posted article count is the total number of articles actually posted. The duplicate count is how many message IDs from posted messages that were found to already exist on the remote system. The missing count is those posted articles that could no longer be found locally, usually because they were canceled locally, or perhaps since they were expired already.

The connection time and the total number of characters transferred via the NNTP layer is logged, in addition to the average speed of article transfer. The speed given is the net rate, the calculation being based on the actual number of characters belonging to news article head and body parts.

A log file may be specified with the -l option, and a folder of posted messages maintained with the -f option.

ARGUMENTS

options: As detailed below.
spoolname: The name of the outgoing spool queue. If not specified, the value of the NNTPSERVER environment variable will be used.
hostname: The hostname of the remote NNTP server to connect to. If not specified, the name of the spool will be used.
port: The port used for connecting to hostname. Will usually not need to be specified, and port "nntp" will be assumed. The port name "telnet" will receive special treatment.

OPTIONS FOR LOGGING

-f folder, --posted folder: Posted articles will be appended to the standard mail folder indicated. Note that the article will be added to the folder the first time posting of an article is attempted. The actual status of the posting will appear in the log file.
-l logfile, --log logfile: Log information about successfully posted articles to the file indicated.
--stat statfile: A summary of article exchange statistics will be logged to the file indicated.
--scanlogs: Obey locks set by the INN scanlogs utility. This is currently only meaningfull for use with a modified scanlogs that understands newsx logs.

OPTIONS FOR NEWS ACCESS

-a authfile, --auth authfile: If specified, the username and password combination found in authfile will be sent to the server as part of the simple authorization protocol when the connection is first made. The file should consist of a line with two blank-separated words, the first being the username, the second the password. Do ensure that this file only has read access only for those users that need to know.
--authgeneric: Authentication with the server is performed using AUTHINFO GENERIC. This works by spawning a child process via /bin/sh string, where string is set in the environment variable NNTPAUTH. This process is expected to handle the authentication portion of the connection.
-r, --reader: A MODE READER command is sent to the remote server. This is used when connecting to remote INN hosts that needs it, and is the default state when using --ihave. With ihave, the command is issued between the ihave and fetch phase, otherwise the command is issued initially.
--no-reader: No MODE READER command is sent to the remote server. This is the default state when not using --ihave.
--readbeforeauth: Send the MODE READER command before attempting to authenticate via an AUTHINFO command. Otherwise, it is sent after the authentication. This option will obviously have no effect unless you also specify both --reader and one of --authgeneric or --auth authfile.
-w chatfile, --chat chatfile: Use chatfile to control any special connect sequence that may be required before the NNTP protocol is connected. Only implemented for the "telnet" protocol.
-x command, --exec command: If specified, the command will be executed after the host port has been opened. This option is used in rare cases when special tricks has to be employed to get an NNTP connection activated.
-y program, --connect program: If specified, connect via a bidirectional pipe to program instead of to a socket. The hostname and port arguments will be ignored.

OPTIONS FOR POSTING

-p, --no-post: Do not attempt to do any posting.
--ihave: Use NNTP IHAVE mechanism for submitting news messages, instead of the default POST. If this is for forwarding news in general, and not necessarily news generated locally, the setting of the --fail and --bounce option should be carefully reviewed.
-m, --no-msgid: Remove any message-ID header field from the posted message. This should cause the receiving end to generate a fresh message-ID. It should never been used together with --ihave.
-k, --keep-path: Keep the Path when posting. In posting mode, the Path is by default stripped together with the NNTP-Posting-Host and Xref header items, so that the path of the news article as posted will appear to start at the actual posting host. With --ihave, the path is always included.
--continue: Ignore (but report) errors about non-existing outgoing spools, and continue with news fetching instead.
--max-path len: All outgoing articles are checked to see if the length of their paths are not too long. This is an extra precaution against inadvertently submitting foreign articles for posting. This option sets the maximum path length allowed. The default value is 1, consistent with a simple, local, newsspool. To disable this feature, set it to an impossibly large value, like 999.

OPTIONS FOR FETCHING

-g, --no-fetch: Do not attempt to fetch any news articles.
-e end_tag, --end end_tag: Specify a tag in the incoming host state file, see in.hosts(5). News fetching will end when a corresponding tag line has been reached. The tag line consists of a colon followed by the tag itself. This is a useful tool for dividing the newsgroups into several categories. Only a few newsgroups may be fetched during prime time, whereas the full list can be fetched once every night, for instance.
-W winsize, --window winsize: Specify the window size with respect to how many outstanding NNTP requests will be allowed at any one time. The default setting is 10. A value of 0 will disable windowing (aka. streaming) completely, making debugging easier. Increasing the window can significantly improve the article transfer speed. The effect is particulary pronounced over slow links and when many of the articles are already present in the local spool, where an improvement in speed of over 10 times has been observed compared to running with windowing disabled. All this at the penalty of an increased host newsserver load, obviously.
-b minspool, --size minspool: Specify the minimum incoming spool size. After every new group, if the spool file has reached the byte size specified, a new spool batch will be made. Setting this value to 1 will cause every newsgroup to be placed in a separate spool.
--rnews: Pipe incoming messages directly to rnews instead of placing them into the in-coming spool.
--pipe-to program: Pipe incoming messages to program instead of placing them into the in-coming spool. A separate program and pipe is initiated for each incoming message.
--rnews-to program: Like --pipe-to, but the pipe is continuous for each session, with article seperation is in rnews style.
-z, --sync: Omit actual pulling of news articles. Instead, simply update the host incoming state file to synchronize with the currently highest available article numbers from the server in question. This option can be useful when a fresh local spool is initiated. No article posting will be attempted. See also in.hosts(5).
-Z, --syncnew: Omit actual pulling of news articles in previously unseen groups only. The net effect is that newly added groups will start fetching from now on, instead of from the first article available at the host server.
--reset: Ignore the latest article information in the host incoming state file and attempt fetching all articles available from the server, subject to the normal history lookup constraints. This option is useful if the remote spool has been reconfigured. See also in.hosts(5).
--filter filterprog: Enable message filtering via the program filterprog. Uses the highwind interface model, where each article is given to the filter in NNTP-fashion, and the filter responds with a 335 to accept, and a 435 to reject a message.
--mfilter filterprog: Enable filtering by message ID. The filterprog will be invoked once for each unseen message, with the message ID as a command line argument. A non-zero return status will prevent the message from being transferred. This function is obsolete, and will probably be removed in later versions.
--maxnew num: Fetch at most num articles from each new and previously unseen newsgroup.
--maxart num: Fetch at most num articles from each newsgroup. Note that this might cause articles to be silently lost.
--no-path: Unless the exclusion pattern is set to newsx, all incoming articles are checked to see if their path is consistent with the exclusions given in /usr/lib/news/sys. This option omits this check.
--forget-inactive: Remove information in the incoming hosts state file about newsgroups that are removed from the /usr/lib/news/active file. See also in.hosts(5).
--minfree N: Don't fetch news if there are less than N kbytes free space in the news spool.
--group list: Specify pattern for groups to be fetched that will apply in addition to the ones in /usr/lib/news/sys. The syntax is basically the same as for INN newsfeeds(5). It is recommended to put list in single quotes, since shells may otherwise treat the ! character in strange ways. For a list that specifies only negations, an initial '*' clause will be assumed.

This option is particularily useful to specify unsymmetric behaviour, i.e. groups where there will be outgoing traffic, but no incoming articles. For instance, to explicitly avoid fetching of control groups, use:
--groups '!control.*'

It can also be useful to specify that only a subset of groups should be: fetched for the occasion, e.g:
--groups 'comp.*,!*.advocacy'
It is recommended to put the argument for --groups in single quotes.

GENERAL OPTIONS

-c, --cnews: Set to C News mode as opposed to INN. Controls details regarding handling of spool and lock files.
-i, --inn: Set to INN mode as opposed to C News. Controls details regarding handling of spool and lock files.
--newline: Uses single newline character as line terminator instead of the carriage return, line feed sequence.
-v, --version: Display program version. Include this when reporting bugs.
--help: Give a very brief usage summary.
--no-ps: Do not update the process status display.

OPTIONS FOR NEWSGROUPS

--list listname: Obtain a list of newsgroups from the newsserver to listname. The list consists of one line per newsgroup, containing the newsgroup name, a blank, and a letter showing the group status (usually 'y' for an active group, 'm' for a moderated group).

: If the first character is a '|', listname is assumed to be a program path which will receive the list as standard input (you will need to put the entire argument in quotes to escape the usual shell interpretation of '|'). If it is a single '-', the list will appear on the standard output. Otherwise it will be assumed to be a filename.

--newlist listname: Obtain a list of newsgroups not currently present on the local server from the newsserver to listname. The list is also subject to the limitations of the sys or newsfeeds file. The operation is otherwise as for the --list option.
--desc descname: Obtain a list of newsgroups descriptions from the newsserver to descname. Only descriptions for locally active newsgroups relevant to the newsfeed in question will be fetched.

: The list consists of one line per newsgroup, containing the newsgroup name, a tab, and a short textual description. A pipe may be specified in descname just as for the --list option.

--alldesc descname: Obtain a list of all newsgroups descriptions available from the newsserver to descname. The operation is otherwise as for the --desc option.

OPTIONS FOR ERROR HANDLING

-t timeout, --timeout timeout: Specify timeout for TCP/IP and lockfile operations, in seconds. The default value is 600.
--fail time: Specify the maximum age of an article. Whenever an attempt of posting an article fails, and the specified time limit is exceeded, the article will be declared as failed, and a bounce message will be generated. The default unit is hours; the suffixes s for seconds, m for minutes, h for hours, d for days and w for weeks would change this. They can also be combined, e.g. "4h30m". In absence of this option, no fail by age will be performed. Setting the time to zero will bounce messages after the first failed attempt.
--bounce addr: Specify the destination address for messages bounced due to the fail time being exceeded. If addr is specified as none, no bounce will be generated. The default value, poster, causes messages to be returned to the sender. This setting should definitely only be used in situations where newsx is utilized for posting locally generated news.
--attach how: Specifies the way the original message is handled for bounced messages:
mime: Send the original as a Mime attachement. This is the default behaviour.
yes: Append the original to the mail message itself.
no: Do not attach the original message.

OPTIONS MOSTLY FOR NEWS GATEWAYS

The following options are not for ordinary use: They would only be used for special configurations where newsx is used as a sort of gateway to inject news messages from other sources.

--inews: Pipe incoming messages directly to inews, inject them into the local spool via the same interface as is used for local new messages.
--inews-options options: Specify command line options for --inews. The leading '-' must be included. The default inews options are "-hOS".
--add-header header: Add a specific header to incoming messages. The header will be added exactly as specified, adding a trailing newline.

: The exception is for the special 4 letter string "Path". This will be replaced by a header line that says "Path: HOSTNAME!not-for-mail" if the incoming message contains no Path:. This can be quite essential for news gateways to prevent injected messages to be retured back to the source.

OPTIONS FOR PROFILE FILES

--profile file: Read a newsx profile from file . This file can contain command line options to specify default newsx behaviour. See newsx.conf(5) for an overview of the format.

Any option may be used in profile files. Some options are really only meaningfull in profile files:

--spoolname spoolname: Specify a default spoolname.
--hostname hostname: Specify a default hostname.
--port port: Specify a default port for connecting to hostname.

OPTIONS FOR CONFIGURATION

Configuration to the local news system peculiarities is traditionally done at build-time, but it is possible to specify at runtime in various ways.

--config: Read and obey the standard news configuration file in /usr/lib/news/bin/config. This allows newsx to adapt to changes in the local newsserver configuration dynamically.
--config-is configfile: Like --config, except that the configuration filename is specified. A list of files may also be specified, seperated by colons. In this case, the first file on this list that exists will be used.
--home newshome: Use newshome as the news home directory instead of the default /usr/lib/news. A full path must be specified.
-s spooldir, --spool spooldir: Use spooldir as the news article spool directory instead of the default /var/spool/news. A full path must be specified.
--togo togofile: Specify the name of the C News togo file for outgoing news. No effect in INN mode. Required only if different from the default value /var/spool/news/out.going/spoolname/togo. A leading '/' specifices a full path, a leading '.' is relative from the current directory, otherwise the name is relative to /var/spool/news/out.going/spoolname.
--batch batchdir: Use batchdir as the outgoing news article spool directory instead of the default /usr/lib/newsbin. A leading '/' specifices a full path, a leading '.' is relative from the current directory, otherwise the name is relative to /var/spool/news.
-h historydb, --history historydb: Specify the name of the history database file, used for checking if news articles are already present in the spool. Required only when using a name different from the default /usr/lib/news/history. An empty argument "" will prevent any referrence to the news history database.
--active activefile: Specify the name of the active newsgroup file for incoming news. Required only if different from the default value /usr/lib/news/active. A leading '/' specifices a full path, a leading '.' specifies a path relative to the current directory, otherwise the name is relative to /usr/lib/news.
--newsfeeds newsfeedsfile: Specify the name of the newsfeed specification file. This will override the default both in C News (sys) and INN (newsfeeds) modes. File name conventions as for --active.
--incoming incomingdir: Use incomingdir as the incoming news article spool directory instead of the default /var/spool/news/in.coming. Fine name conventions as for --active, except that the default directory is /var/spool/news.
--inhosts inhostsdir: Use inhostsdir as the news host active file directory instead of the default /var/spool/news/in.hosts. Name conventions as for --incoming. See also in.hosts(5).

OPTIONS FOR TESTING AND DEBUGGING

--debug level: Set debug level. Any level different from zero diverts reporting to stderr instead of syslog, and turns on extra debugging output. Level 2 will output a '.' for every new article. Level 3 will log all NNTP commands. Level 4 will produce the same state information that is available via the process status display. Going to level 5 will produce additional debug information, level 6 will include history database debug, while level 7 also will show all communications over the NNTP socket, article content included.
-d, --verbose: Enable debugging, and sets the debug level according to the number of times it is specified.
-n, --dry-run: No-action flag, will "fake" an NNTP connection, and leave the outgoing batch untouched. Probably useful only with --debug for debugging and dry-testing.
--verify: Used together with --configor --config-is, causes the configuration to be verified, but not adapted to.
-o, --keepold: Keep the previous outgoing spool in an ".old" file. This might seem like a useful option for ordinary use, but the problem is that a non-empty file might cause a news watch program to believe there is a stale outgoing spool.
-u, --no-force: After the timeout of lockfile operations, just give up and do not attempt to unlock the lockfiles and remove the stale process.
--no-queue: Do not queue up for access to a news host. If a lock already exists for the specified host, just give up and don't queue up for it.
--locks locksdir: Use locksdir as the lock file directory instead of the default /usr/lib/news. Fine name conventions as for --active, except that the default directory is /usr/lib/news. If the string none is supplied, no locking will occur. All this should of course be used only if you really, really know what you are doing.
-q "msgid", --enquire "msgid": Enquires whether the msgid is present in the local history database. The msgid should include the angle brackets. Implies --no-post and --no-fetch.
--no-hostlock: Do not implement the newsx host access lock. Should only be used if you really know what you are doing.
--no-next: Do not use the NNTP NEXT command. This causes newsx not to attempt to use NEXT for filling out gaps in the article sequence.
--missing "num": Tunes the number of missing articles in a row before a NEXT command is issued, instead of a sequential STAT. A value of zero will cause NEXT to be used extensively. The default value is 0 if no window, 2 if a window is specified.
--keep-fake: Some remote news servers generate faked news articles. Normally, you would want newsx to skip them, but with this option you can actually fetch them.

SPECIAL CONNECT SEQUENCE

The -w option specifies that a special connect sequence is required, and that a script for this sequence can be found in the chatfile. This file consists of lines that contains pattern receive and send pairs, separated by blanks. The patterns may be enclosed in quotes. The script is currently only available for the "telnet" protocol.

A typical invocation line would be:

: newsx -w chat.acme acme login.acme.net telnet

The file chat.acme could look like this:

:        # login for acme.net NNTP

       login: myusername

       Password: mypassword

       $ "exec telnet news.acme.net nntp"

       ECHO

The script will have to be adapted for local conditions, of course.

The special tag ECHO specifies that line echo should be suppressed, required since the host telnet operates in line mode.

Since this file usually contains passwords and other sensitive information, ensure that read access to it is limited.

PROCESS STATUS

The ps process status will also show the current newsx status. During the actual news transfer phase, it will show which group and article number that is being fetched. A continuous status can be obtained by;

: newsq -c

This feature can be disabled via the --no-ps option.

CONFIGURATION FORMAT

For INN, the following configuration items will be used:

$INND: Signals an INN configuration.
$NEWSBIN: Directory for INN binaries.
$NEWSHOME: News home directory.
$PATHETC: Directory for news configuration files. Defaults to $NEWSHOME if not available.
$SPOOL: If $SPOOL contains the string "/articles", the incoming host directory $INHOSTS will be formed by replacing it with "/inhosts". Otherwise, "$SPOOL/in.hosts" will be used.
$BATCH: Outgoing spool.
$INCOMING: Incoming spool.
$ACTIVE: Active file.
$HISTORY: History database.
$LOCKS: Directory for lockfiles.
$NEWSFEEDS: Newsfeeds file.
$EXTENDEDDBZ: True is history database in extended format.
$STORAGEAPI: True if storage-API is active.

For C News:

$NEWSBIN
$NEWSCTL: Equivalent to INN $NEWSHOME. $ACTIVE will be "$NEWSCTL/active". $HISTORY will be "$NEWSCTL/history". $NEWSFEEDS will be "$NEWSCTL/sys". $LOCKS and $PATHETC will be "$NEWSCTL".
$NEWSARTS: Equivalent to INN $SPOOL. $BATCH will be "$NEWSARTS/out.going". $INCOMING will be "$NEWSARTS/in.coming". $INHOSTS will be "$NEWSARTS/in.hosts".

LOG FILE FORMAT

The -l log file of posted articles will contain one line for each article. To be compatible with the INN format, each field is separated by a tabstop:

Month: Standard 3 letter abbreviation.
Date: 2 digits
HH:MM:SS: Local time.
Year: 4 digits.
Spool: Name of outgoing spool.
<Msgid>: Within angle brackets.
Filename: As it appears in the spool file.
Sender: The value of the Sender-field, or From-field if no Sender is available.
Status: "OK" for successfully posted items, "Duplicate" for items already posted.
Lines: Number of lines in the posting.
Status: Message status. "OK" if everything went well, otherwise an error message appears.

INCOMING SPOOL FORMAT

The incoming spool consists of files with the file name ending by ".t". Each file can contain many articles. Each article is prefixed by the following header:

: #! rnews N

The number N is the number of bytes of the article proper, not counting the header line. The article follows after the header line. It appears exactly as received from the news server except that CR/LF sequences are replaced by proper newlines, and the terminating full stop is not included.

MESSAGE HEADER HANDLING

Incoming message headers will pass through without modification, with the exception that the text "newsx" will be added if not there already when the exclusion pattern for the newshost is set to "newsx".

Most outgoing message header items will also pass through, altough some header items are handled specially:

From:: Will be used for indentifying the sender if everything else fails.
Message-ID:: Will be used for logging purposes. The --no-msgid will remove this item for messages being posted.
NNTP-Posting-Date:
NNTP-Posting-Host:: Will always be removed for messages being posted.
Path:: Will be removed for messages being posted, unless the --keep-path option is given.
Reply-to:: Will be used for indentifying the sender.
Sender:: Will be used for indentifying the sender.
Xref:: Will always be removed from outgoing messages.
X-Server-Date:
X-Trace:
X-Complaints-To:: Will always be removed for messages being posted.

DIAGNOSTICS

newsx returns error codes as follows:

: 0 - Successful completion
1 - General system error
2 - Incorrect arguments supplied.
3 - Error connecting to remote host
4 - NNTP Protocol error
5 - Errors accessing outgoing spool file.
6 - Errors accessing host newsrc file.
7 - Errors accessing incoming spool file.
8 - Errors accessing local active and sys files.
9 - Socket access error (e.g. connection timeout)
10 - Errors accessing authinfo file.
11 - Connect script failure.
12 - Option -q did not find the message ID.
13 - Wait for lock timed out.

NOTES

Distribution fields are ignored by newsx, and the handling of them left to the news server.

BUGS

Please report any bugs to the Bugzilla.

FILES

AUTHOR

Egil Kvaleberg <egil@kvaleberg.no>

Index

NAME
SYNOPSIS
DESCRIPTION
OPERATION
ARGUMENTS
OPTIONS FOR LOGGING
OPTIONS FOR NEWS ACCESS
OPTIONS FOR POSTING
OPTIONS FOR FETCHING
GENERAL OPTIONS
OPTIONS FOR NEWSGROUPS
OPTIONS FOR ERROR HANDLING
OPTIONS MOSTLY FOR NEWS GATEWAYS
OPTIONS FOR PROFILE FILES
OPTIONS FOR CONFIGURATION
OPTIONS FOR TESTING AND DEBUGGING
SPECIAL CONNECT SEQUENCE
PROCESS STATUS
CONFIGURATION FORMAT
LOG FILE FORMAT
INCOMING SPOOL FORMAT
MESSAGE HEADER HANDLING
DIAGNOSTICS
NOTES
BUGS
FILES
AUTHOR
SEE ALSO

This document was created by man2html, using the manual pages.
Time: 22:21:56 GMT, July 26, 2000