SCOUG-HELP Mailing List Archives

<< Previous Message << >> Next Message >>

Date:	Tue, 29 Jul 2003 06:27:42 PDT7
From:	Peter Skye <pskye@peterskye.com >
Reply-To:	scoug-help@scoug.com
To:	scoug-help@scoug.com
Subject:	SCOUG-Help: utility to convert records to lines ?

Content Type: text/plain

=====================================================
If you are responding to someone asking for help who
may not be a member of this list, be sure to use the
REPLY TO ALL feature of your email program.
=====================================================

Steven Levine wrote:
>
> the loop logic can probably be better written as:
>
> c = chars('inFile')
> b = charin('inFile',,c)
> call stream 'inFile', 'C', 'CLOSE'
> do i = 1 to c by 83 ...
>
> It should be measurably quicker.

Okay, I decided to measure it. :)

The results (on my system, with all the vagaries of
drive hardware buffers, HPFS cache and REXX internal
I/O buffering) show that for *typical* applications
there's a noticeable speed increase if, as you suggest,
you first read the entire file into a buffer and then
separate the records using substr().

But, curiously, for very large record sizes this method
is slower.

Below are my results. Note that:
- "Individual Records" uses charin() to read each record.
- "Buffered Records" uses substr() to separate the records.

I calculated microseconds/record because the resulting
values are easy to compare. This value is in parentheses.

I ran the benchmark about 10 times. The following results,
from the last run, are typical of all of the other runs.

---- Seconds (uSec/Rec) to Read ---- Speed
File Size Record Size Individual Records Buffered Records Ratio
========== =========== ================== ================ =====

1,000,000 10 6.18 ( 61.8) 1.81 ( 18.1) 3.41
1,000,000 100 0.71 ( 71.0) 0.27 ( 27.0) 2.63
1,000,000 1,000 0.13 ( 130.0) 0.10 ( 100.0) 1.30
1,000,000 10,000 0.10 (1000.0) 0.12 (1200.0) 0.83

4,000,000 10 28.22 ( 70.6) 8.56 ( 21.4) 3.30
4,000,000 100 3.62 ( 90.5) 2.09 ( 52.3) 1.73
4,000,000 1,000 1.28 ( 320.0) 1.46 ( 365.0) 0.88
4,000,000 10,000 1.14 (2850.0) 1.58 (3950.0) 0.72

16,000,000 10 112.55 ( 70.3) 33.66 ( 21.0) 3.34
16,000,000 100 14.79 ( 92.4) 8.40 ( 52.5) 1.76
16,000,000 1,000 5.07 ( 316.9) 5.74 ( 358.8) 0.88
16,000,000 10,000 4.81 (3006.3) 5.95 (3718.8) 0.81

64,000,000 10 445.14 ( 69.6) 134.91 ( 21.1) 3.30
64,000,000 100 57.91 ( 90.5) 33.40 ( 52.2) 1.73
64,000,000 1,000 18.88 ( 295.0) 21.43 ( 334.8) 0.88
64,000,000 10,000 16.28 (2543.8) 22.71 (3548.4) 0.72

Here's the applicable Rexx code:

/* Benchmark using CharIn() to read each record. */
call OpenInputFile TestFile
call Time( "R" )
do while chars( TestFile ) >= iRecSize
x = charin( TestFile, , iRecSize )
end
ElapsedTimeByRecord = Time( "E" )
call CloseInputFile TestFile

/* Benchmark using CharIn() to read the entire file into a buffer. */
call OpenInputFile TestFile
call Time( "R" )
iFileSize = chars( TestFile )
buffer = charin( TestFile, , iFileSize )
do Index = 1 to iFileSize by iRecSize
x = substr( buffer, Index, iRecSize )
end
ElapsedTimeByBuffer = Time( "E" )
call CloseInputFile TestFile

If anyone wants to see the entire program, ask on the
SCOUG-Programming list and I'll post it.

- Peter

=====================================================

To unsubscribe from this list, send an email message
to "steward@scoug.com". In the body of the message,
put the command "unsubscribe scoug-help".

For problems, contact the list owner at
"rollin@scoug.com".

=====================================================

Return to [ 29 | July | 2003 ]

SCOUG, Warp Expo West, and Warpfest are trademarks of the Southern California OS/2 User Group. OS/2, Workplace Shell, and IBM are registered trademarks of International Business Machines Corporation. All other trademarks remain the property of their respective owners.