SCOUG-Programming Mailing List Archives

<< Previous Message <<

Date:	Sun, 2 Jul 2000 23:56:37 PST
From:	Peter Skye <pskye@peterskye.com >
Reply-To:	scoug-programming@scoug.com
To:	scoug-programming@scoug.com
Subject:	SCOUG-Programming: Call for programmers - adopt-an-app lives

Content Type: text/plain

Steven Levine wrote:
>
> >Can News Harvest download multiple groups simultaneously?
>
> Not directly. You can set it up to run multiple instances to get a
> similar effect. More or better multithreading would be high on the list
> of additions.

Multiple instances is not a good idea. You need some kind of front-end
logic with a database that manages the groups and decides how often to
check for new files (i.e. prioritizes the downloads). Multiple
instances require that you split your newsgroup list into sub-groups and
assign one sub-group to each running instance of the program --
workable, but not a "best fit".

To maximize throughput, _everything_ has to _finish_ at the _exact_ same
time. You have to control the priorities of your parallel
threads/instances so that everything finishes simultaneously. Tasks
that get ahead of the others have to get their priorities reduced so
everything marches in lock step towards completion. If there are other
considerations, such as trying to download your warez files before the
news server deletes them, then you have a milepost at which a certain
subset (such as "all messages older than 48 hours") has to be
completed. But your _goal_ is for everything to be x% complete at any
given time, where x is the _same_ for every newsgroup. If implementing
this becomes unwieldy, then you have to download in blocks (i.e.
download everything older than 48 hours, then make a second pass and
download everything older than 24 hours, etc.).

By the way, forcing all parallel tasks to maintain an "equal percentage
done" position forces the tasks that cause the bottlenecks to the top of
the priority list. Somewhere around here I have an unpublished paper
about this -- "somebody" told me no one was interested in such
mathematical treatises, so I never submitted it for publication anywhere
. . .

- Peter

=====================================================

To unsubscribe from this list, send an email message
to "steward@scoug.com". In the body of the message,
put the command "unsubscribe scoug-programming".

For problems, contact the list owner at
"rollin@scoug.com".

=====================================================

Return to [ 02 | July | 2000 ]

SCOUG, Warp Expo West, and Warpfest are trademarks of the Southern California OS/2 User Group. OS/2, Workplace Shell, and IBM are registered trademarks of International Business Machines Corporation. All other trademarks remain the property of their respective owners.