Download
PipeLib Class Library
Introduction
CMS/TSO PipeLines is a product that runs on IBM mainframes. On a z/VM system it runs uner CMS. On
a z/OS system it runs under TSO. As the name suggests data is passed thru a series of builtin or
user defined stages. Along the way the data can be modified or deleted from the pipe. You may be
saying to yourself there is nothing new in that. What is different is that PipeLines supports
concurrent parallel pipes and data can be routed into and out of a pipe.
System Requirements
While the PipeLib class library can be accessed from a .Net program, I think you will find
that running it under PowerShell more useful. I do all of my testing through PowerShell and
any pipelines I supply will most likely be PowerShell scripts. Therefor as minimum system requirement's
I recommend the following.
PowerShell V1.0
.Net Framework V2
Invoking a User Stage
A user stage is invoked by the PS built in stage. This stage takes as
parameters the name of the user written stage and any parameters the user wishes to pass
to the user stage. The basic outline of a user stage is as follows:
#BasicStage.ps1
1)param ([object] $usp, [String] $parms)
2) ... Perform any needed setup, such as parsing the passed parameter string ...
3) usp-peekto 'inputRec'
4) while ($rc -eq 0) {
5) ... process the record
6) usp-output 'outputRec'
if ($rc -eq 0) {
7) usp-ReadTo
8) usp-PeekTo 'inputRec'
}
}
Line 1 is the only line that is
mandatory. It must be present in every script that uses
any of the user stage commands. Lines 3-8 sets up the basic data flow needed to keep the data
records in sequence. This is not enforced and the user is free code what ever is best for the
application.
The $usp variable defined in line 1 is the interface that connects the script to the
PipeLib class library. The pipe user commands check for the presence and the
validity of this variable.
Pipe User Commands
Below is the list of commands available in a pipe user stage.
- Run-CallPipe
- This command executes a pipeline in parallel with
the current pipeline. The user stage is suspended while the new pipeline runs. The user stage resumes
when the CallPipe command ends. The string that defines the new pipeline is passed as an
argument to this command.
Examples: run-callpipe $pipe or run-callpipe "pipe definition" - Run-AddPipe
- This command executes a pipeline in parallel with
the current pipeline. Unlike the CallPipe command the user stage continues to run. The string
that defines the pipeline is passed as an argument to this command.
Examples: run-addpipe $pipe or run-addpipe "pipe definition" - Usp-PeekTo
- This command reads without consuming the next record passed to the
stage. The name of a variable to receive the record is passed as an argument.
Examples: usp-peekto "varname" - Usp-ReadTo
- This command reads and consumes the next record passed to the
stage. The name of a variable to receive the record is an optional argument.
Examples: usp-readto "varname" or usp-readto - Usp-Output
- This command writes data to the currently selected output stream.
The output data is passed in a variable or a literal string.
Examples: usp-output $line or usp-output "literal data string" - Usp-Select
- This command selects the data stream to used for input and/or output.
The primary input and output are selected by default when a user stage is started.
This command requires two parameters:
- The type of stream to process: "Input" or "Output"
- Stream number or identifier.
Examples: usp-select "both" 0, usp-select "input" 1, usp-select "input" "stream-name" - Usp-Commit
- This command sets the commit level this stage is to be set to.
The new commit level is a required parameter.
Examples: usp-commit 0 - Usp-NoCommit
- This command prevents the user stage from auto committing.
Examples: usp-nocommit - Usp-ScanRange
- This command scans a data string for a location range parameter.
The command takes four parameters:
- Indicates if the location parameter is required or optional and assumes one of the following
values: Required or Optional
- A variable name to receive the generated range location token.
- The name of a variable to receive the remainder of the data string after the location has been
parsed off.
- The fourth parameter is optional. It specifies data to be parsed. If omitted the parameter string
passed to the user stage is used.
Examples: usp-scanrange "required" "token" "rest" "fs - f1", usp-scanrange "optional" "token" "." - Usp-GetRange
- This command extracts the data specified by a previous ScanRange from a data record.
The command takes four parameters:
- The token generated by a previous call to ScanRange
- The type of variable to receive the extracted data. It can assume one of the following values:
"VARiable" or "STEM".
- The name of a variable to recieve the extracted data.
- The data record from which the data is to be extracted.
Examples: usp-getrange $token "var" "varname" $data, usp-getrange $token "stem" "stemname" $data - Usp-ScanString
- This extracts the next word or delimited string
from a data string. The command takes three parameters:
- The name of the variable to receive the string.
- The name of a variable to receive any residual data.
- The data string from which the string is to be extracted. This parameter is optional.
If omitted the parameter string passed to the userstage is used.
Examples: usp-scanstring "string" "rest" $data, usp-scanstring "string" "." - Usp-Short
- This command shorts the currently selected input
stream to the currently selected output stream.
Examples: usp-short - Usp-Sever
- This command severs the currently selected input or
output stream. The single parameter specifies which stream to sever. If the severed stream was
redirected by a CallPipe or AddPipe connector, Then an attempt is made to restore the
original connection.
Examples: usp-sever "input" or usp-sever "output" - Usp-StreamNum
- This command gets the stream number of the
specified stream. The command takes two parameters:
- The type of stream to process: "Input" or "Output"
- Stream number or identifier. This parameter is optional. If omitted the currently slected input
or output stream is used.
If the stream is connected the $rc variable will contain the stream number. If not connected it will contain a -4.
Examples: usp-streamnum "input" 1, usp-streamnum "output" "id" - Usp-StreamState
- This command queries the status of the specified
stream. The command takes one or two parameters:
- When the first parameter is "Input" or "Output", the second parameter, if specified, is the number,
or id, of the stream to query. The default is the currently selected stream.
- When the first parameter is "Summary", a return code of 0 indicates that at least one input and
one output stream are connected.
- When the first parameter is "All" then the second parameter is the name of variable to recieve
the status of each pair of streams.
Examples: usp-streamstate "input" 1, usp-streamstate "all" "varname" - Usp-MaxStream
- This command returns the max stream number for
the selected stream type The command takes one parameters:
- The type of stream to process: "Input" or "Output"
Examples: usp-maxstream "input", usp-maxstream "output"
- Usp-AddStream
- This command adds an unconnected stream to the
user stage. The command takes two parameters:
- The type of stream to process: "Input", "Output", or "Both"
- Stream identifier. This parameter is optional. If omitted then use Usp-MaxStream to get the number
of the added stream.
Examples: usp-addstream "input" "strmid", usp-addstream "both"
AddPipe and CallPipe
Of the above commands CallPipe and AddPipe require special attention. These commands
insert a parallel pipeline into the existing pipe line and depending on the options coded for parallel
pipe it may or may not interact with the existing pipeline. The pipe definition syntax for the
CallPipe and AddPipe commands contain one additional element over that of the
Run-Pipe command.
┌─<─endchar┐
>>─┬AddPipe─┬┬─────────────────────────┬┬─────────┬┴┤ PipeLine├┴─><
└CallPipe┘│ ┌─<─────────────────┐ │├stagesep─┤
└(─┴┬┬STAGESEP──┬xorc─┬┴)─┘└endchar──┘
│└SEPerator─┘ │
├ENDchar xorc─────┤
├ESCape xorc──────┤
├NAME─pipename────┤
└TRACE────────────┘
Group PipeLine
├─┬───────────┬─>
└┤connector├┘
┌─<─stagesep───────────────────────────────────────────────┐
>─┴┬────────────────────────────────────────────────────────┬┴─>
├┬────────────┬┬───────────────┬stagename─┬─────────────┬┤
│└┤LabelGroup├┘└(─┬───┬TRACE─)─┘ └stageoptions─┘│
│ └NO─┘ │
└┤LabelGroup├────────────────────────────────────────────┘
>─┬───────────┬─┤
└┤connector├┘
Group <a id="""Connector""">Connector</a>
├─*─┬───────────────────────────┬:──┤
│ ┌.*────────────┐│
└.─┬INPUT──┬┼──────────────┼┘
└OUTPUT─┘└.─┬*─────────┬┘
├streamnum─┤
└streamid──┘
Group LabelGroup
├─┬────────────────┬─┤
├label:──────────┤
└label.streamid:─┘
- Options
- STAGESEP xorc
SEPerator - Specify the character to be used to separate the stages of a pipe. It can be specified
as a single character or as the hex digits of a character. The default separator is the
virtical bar, "|".
- ENDchar xorc
- Specifiy the character that separates the pipes of a pipeline. It can be specified
as a single character or as the hex digits of a character. There is no default.
- ESCape xorc
- Define an escape character that can be used to override the processing of a
character that has special meaning to the PipeLine parser. Any character that follows the
escape character is treated as an ordinary character. There is no default.
- NAME pipename
- Specify a name to be associated with this pipeline.
- TRACE
- Turn on tracing for the pipeline.
- Pipeline
- stagesep
- The stages of a pipe are separated by the STAGESEP character. The default is "|".
- endchar
- If there are multiply pipes in the pipeline, separate them by the ENDCHAR.
- NO
- If tracing is turned on for the pipeline then "NO TRACE" will turn it off for this
stage of the pipeline.
- TRACE
- Turn tracing on for this stage of a pipeline. If prefixed by the "NO" keyword then
tracing is turned off for this stage.
- stagename
- The name of the stage to be executed.
- stageoptions
- Supply any options needed by the specified stagename.
- *
- Identifies this as the start of a connector definition.
- INPUT
- This connector applies to an input stream of the user stage. If the connector is the
first stage of a pipe then the user stages input stream is directed to the first stage
of the CallPipe or AddPipe pipeline. If the connector is the last stage of a pipe then
the output of the CallPipe or AddPipe pipeline is directed to the user stage input.
- OUTPUT
- This connector applies to an output stream of the user stage. If the connector is the
first stage of a pipe then the user stages output stream is directed to the first stage
of the CallPipe or AddPipe pipeline. If the connector is the last stage of a pipe then
the output of the CallPipe or AddPipe pipeline is directed to the user stage output.
- *
- This connector applies to the currently selected input or output stream.
- streamnum
- The stream number of the effect user stage data stream.
- The stream ID of the effected user stage data stream.
- :
- Marks the end of a connector definition.
- label:
label.streamid: - A label that identifies additional input and/or output data streams for the stage it is specified on. The first occurrence of a label is called a label definition. It establishes connection point where other stages, in other pipes, receive data from or send data to. Each subsequent use of the same label is called a label reference. Use label references to define additional input and output streams for the stage. To use a label reference, specify a stage containing only a label with no stagename.
A streamid assigns a symbolic name to a stream. Name a stream by adding an identifier to the label. Write the label immediately followed by a period (.) and up to 4 alphabetic characters or a combination of alphabetic characters and digits that includes at least one alphabetic character. A stream identifier must be immediately followed by a colon with no intervening blanks.
The
Connector element is used to redirect the data streams of the user stage to the pipeline defined by the
CallPipe or
AddPipe.
Consider the pipe "A|Z|D" where "Z" is a user stage that issues a CallPipe or AddPipe.
The following demostrates the effect of connectors on data flow through a pipeline when CallPipe is used.
CallPipe: B|C *.Input:|B|C B|C|*.Output: *.Input:|B|C|*.Output:
A A A A
│ B └─B │ B └─B
DataFlow: Z │ Z │ Z │ Z │
│ C │ C C C
D D D─┘ D─┘
Restorable: NA Yes Yes Yes
(1) (2) (3) (4)
- With no connectors, data flows from one stage to the next in each pipeline.
- A's output stream is directed to B's input. Z's input stream is disconnected.
- C's output stream is directed to D's input. Z's output stream is disconnected.
- A's output stream is directed to B's input. C's output stream is directed to D's input.
Z's input and output streams are disconnected
The following demostrates the effect of connectors on data flow through a pipeline when AddPipe
is used.
AddPipe: B|C *.Input:|B|C B|C|*.Output: *.Output:|B|C B|C|*.Input:
A A A A A B
│ B └─B │ B │ │
DataFlow: Z │ Z │ Z │ Z Z─ C
│ C │ C ┌─C └─B │
D D D D │ D
C
Restorable: NA No No Yes Yes C
(1) (2) (3) (4) (5)
AddPipe: *.Input:|B|C|*.Output: *.Input:|B|C|*.Input:
A A
└─B └─B
DataFlow: Z │ │
C C
D─┘ Z─┘
│
D
Restorable: No No
(6) (7)
AddPipe: *.Output:|B|C|*.Input: *.Output:|B|C|*.Output:
A A
┌─┐ ┌─B │ ┌─B
DataFlow: │ Z │ │ Z │ │
│ └─┘ C └─┘ C
└─────┘ ┌───┘
D D
Restorable: Yes No
(8) (9)
- With no connectors, data flows from one stage to the next in each pipeline.
- A's output stream is directed to B's input. Z's input stream is disconnected.
- C's output stream is directed to D's input. Z's output stream is disconnected.
- Z's output stream is connected to B's input. D's input stream is disconnected.
- C's output stream is connected to Z's input. A's output is disconnected.
- A's output stream is directed to B's input. C's output stream is directed to D's input.
Z's input and output streams are disconnected
- A's output stream is connected to B's input. C's output stream is connected to Z's
input.
- Z's output stream is connected to B's input. C's output stream is connected to Z's input.
A's output and D's input streams are disconnected. All stall is possible.
- Z's output stream is connected to B's input. C's output stream is connected to D's input.
An Example
Below is an example of a pipe that invokes a user stage.
#AddPipeDemo
$data = "B999 1323 FFFF BA82 1A43 20DD"
$pipe = 'var data'+
'|split'+
'|PS userstage'+
'|stem Stem1'
run-pipe $pipe
The user stage called by the above pipeline is:
# AddPipeDemo userstage
1) param ([object] $usp, [String] $parms)
# Read parameter file
2) run-addpipe '< addpipe.conf | *.input:' # Connect input to file
3) usp-nocommit # Disable automatic commit
4) usp-readto 'line' # Read first line of file
while ($rc -eq 0) { # Process all lines
if ($line[0] -ne '#') { # Ignore any comments
usp-scanstring 'sortOrder' '.' $line
write-host "The selected sort order is: $sortOrder"
}
usp-readto 'line' # Read next line
}
$order = 'fa-ff'
if ($sortOrder -eq 'Lower') {
$order = '0a-0f'
}
5) usp-sever 'input' # Re-instate input file
6) usp-commit 0
7) $pipe = '*.input:' + # connect to output of stage preceding caller
'|xlate 1-* A-F '+$order+ # Translate A to F
'|sort' + # Sort the records
'|xlate 1-* '+$order+' A-F'+ # Restore original characters A to F
'|*.output:' # connect to input of stage following caller
8) run-callpipe $pipe
- The mandatory parameter definitions requied by a user stage.
- Initiate the pararell pipe. The output of this pipe is directed to the
primary input of this stage. The previous input stream is svaed for a posible
reconnection later.
- This stops the user stage from doing an Auto Commit. All stages start at some commit level. For user stages it is -1. Most of the builtin stages start at some level, verify the state of the stage and then commit to level 0. Stages at the lowest commit level run before stages at a higher commit level. All builtin stages will eventually end up at commit level 0. Thus with this command this stage will stay at level -1 until it explicitly commits to a higher level. Normally any IO request by a user stage causes an Auto Commit to level 0.
- Read the first input record. Note that this read will consume the record. There is no need to worry about record sequence because the records never leave this stage.
- After all records have been processed we restore the original input stream.
- Here we commit to level 0. The other stages will have already commited to level 0 and are waiting for this stage to join them. Once all stages are at level 0 they are all eligible to run.
- Define a pipeline to be executed.
- Execute the pipeline.
Another Example
The following example of a user stage takes records that conatin binary data and dumps them
in hex format to the screen.
#Dump.ps1
param ([object] $usp, [string] $parms)
usp-scanstring 'width' '.'
if ($width -eq '16') {
$text = ' 1-* 45 '
} else {
$width = '32'
$text = ' 1-16 83 17-* 101 '
}
$pipe = '(end ?) *.Input:'+
'|deblock fixed '+$width+
'|f:fanout'+
'|copy'+
'|s:spec set (#1=1; #2=8; #3=0)'+
' while (#1<length() && #3<4) do'+
' print (c2x(substr(record(), #1, 4))) (#2, 8)'+
' set (#1+=4; #3+=1; #2+=9)'+
' done'+
' set (#2=45; #3=0)'+
' while (#1<length() && #3<4) do'+
' print (c2x(substr(record(), #1, 4))) (#2, 8)'+
' set (#1+=4; #3+=1; #2+=9)'+
' done'+
' select 1 '+$text+
' pad 0 print (d2x(#0)) 1.4 r'+
' set (#0+='+$width+')'+
'|*.output:'+
'? f:'+
'|xlate 00-1f . 80-9f .'+
'|s:'
run-callpipe $pipe
An sample of the output produced by this stage, called with a width parameter of 16:
0000 3f523f4e 54465320 20202000 02080000 ?R?NTFS .....
0010 00000000 003f0000 3f003f00 043f0000 .....?..?.?..?..
0020 00000000 3f003f00 11583f03 00000000 ....?.?..X?.....
0030 3f7f0c00 00000000 3f024000 00000000 ?⌂......?.@.....
References
z/VM V5R3.0 Pipelines Reference
z/VM V5R2.0 CMS Pipelines User's Guide
CMS/TSO Pipelines Runtime Library Distribution
Author's Help Files
