I am attempting to debug a small file tailoring job but don't get any output when attempting to use the trace command ISPFTTRC which is described at
https://www.ibm.com/support/knowledgecenter/SSLTBW_2.1.0/com.ibm.zos.v2r1.f54dg00/ispfttrc.htm#ispfttrc
The REXX interprets without warnings.
I have also tried coding the command as
"ISPFTTRC LIST"
but this did not work.
I'd like to:
Get the ISPFTTRC command to return some output since it will be useful for future debugging.
Fix the problem with the job.
Any help would be much appreciated, thanks.
Here is the JCL:
//DOIT EXEC PGM=IKJEFT01,
// PARM='ISPSTART CMD(NDRACMQ)'
//SYSPROC DD DISP=SHR,DSN=&SYSUID..ALL.REXX
//* --- ISPF FILES ---
//ISPSLIB DD DISP=SHR,DSN=&SYSUID..ALL.SKELS
//* --- OUTPUTS ---
//ISPFILE DD DISP=SHR,DSN=&SYSUID..ALL.SKELS(NDSACMO) tailored output
//ISPFTTRC DD DISP=SHR,DSN=&SYSUID..ISPFTTRC debugging output
//* --- MORE ISPF FILES ---
//ISPPROF DD DSN=&&ISPPROF,DISP=(NEW,DELETE,DELETE),
// LRECL=80,RECFM=FB,DSORG=PO,SPACE=(TRK,(2,2,2))
//ISPPLIB DD DISP=SHR,DSN=&SYSUID..ALL.PARMLIB needed? I think not
//ISPTLIB DD DISP=(NEW,DELETE,DELETE),DSN=&&ISPTLIB,
// SPACE=(TRK,(1,1,1)),LIKE=SYS1.SYSTLIB
// DD DISP=SHR,DSN=SYS1.SYSTLIB
//ISPMLIB DD DISP=SHR,DSN=SYS1.SYSMLIB
//* --- OTHER FILES ---
//SYSPRINT DD SYSOUT=*
//ABNLDUMP DD DUMMY
//SYSTSIN DD DUMMY
//SYSTSPRT DD SYSOUT=*
The REXX (simplified):
/* REXX - Bulk ACM queries */
ISPFTTRC LIST
"ISPEXEC FTOPEN"
ele=aaaaa
"ISPEXEC FTINCL NDSACMQ"
ele=bbbbb
"ISPEXEC FTINCL NDSACMQ"
"ISPEXEC FTCLOSE"
exit
The skeleton code from &SYSUID..ALL.SKELS(NDSACMQ):
)CM *-----------------------------------------------------------------*
)CM * Skeleton SCL used to create bulk ACM queries. *
)CM *-----------------------------------------------------------------*
LIST USING COMPONENTS FOR
ELEMENT &ELE ENVIRONMENT *
SYSTEM * SUBSYSTEM *
TYPE * STAGE NUMBER *
OPTIONS
.
Output was going to a dynamically allocated file &SYSUID.**.ISPFT.TRACE.
The way to code the ISPFTTRC command in the REXX was simply "ISPFTTRC" without the quotes, this gives output.
Checking the RC from the command within the REXX was helpful.
I will be able to resolve the problem with the job now that I have trace output.
Related
I need to create a batch job for searching a data set list; I need to submit the job over FTP.
I am a complete newbie in JCL, so could anyone help me with an example job with SRCHFOR-command?
I need to also create a job for the IDCAMS utility command Listcat
I would be really thankful only for an example job.
Given you indicated you don't know where to start here is a sample job that will run the search in the first step and run IDCAMS to execute a Listcat in the second step.
This link has some useful information on SuperC.
Note: Your jobcard will need to be customized based on your installation's requirements. This job can be submitted via FTP and the subsequent output can be retrieved via FTP as well.
//SRCHCMP JOB (CCCCCCCC),'HOGSTROM',
// MSGLEVEL=(1,1),
// MSGCLASS=O,
// CLASS=A,
// NOTIFY=&SYSUID
//*
//SRCHFOR EXEC PGM=ISRSUPC,PARM=('SRCHCMP,ANYC')
//NEWDD DD DSN=USER1.TEST.CNTL,DISP=SHR
//OUTDD DD SYSOUT=*
//SYSIN DD *
SRCHFOR 'NEWDD'
/*
//*
//IDCAMS EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
LISTCAT ENTRY('USER1.TEST.CNTL') ALL
/*
//
Output from the first step will look like this:
ISRSUPC - MVS/PDF FILE/LINE/WORD/BYTE/SFOR COMPARE UTILITY- ISPF
FOR z/OS 2019/02/26 22.45 PAGE 1 LINE-#
SOURCE SECTION SRCH DSN: USER1.TEST.CNTL
SRCHFOR --------- STRING(S) FOUND
-------------------
8 //NEWDD DD DSN=USER1.TEST.CNTL,DISP=SHR
11 SRCHFOR 'NEWDD'
ISRSUPC - MVS/PDF FILE/LINE/WORD/BYTE/SFOR COMPARE UTILITY- ISPF
FOR z/OS 2019/02/26 22.45 PAGE 2
SEARCH-FOR SUMMARY SECTION SRCH DSN: USER1.TEST.CNTL
LINES-FOUND LINES-PROC MEMBERS-W/LNS MEMBERS-WO/LNS COMPARE-COLS
LONGEST-LINE
2 4436 1 41 1:80 80
PROCESS OPTIONS USED: ANYC
THE FOLLOWING PROCESS STATEMENTS (USING COLUMNS 1:72) WERE PROCESSED:
SRCHFOR 'NEWDD'
Output from the IDCAMS Listcat looks like this:
IDCAMS SYSTEM SERVICES TIME: 22:45:34 02/26/19 PAGE 1
LISTCAT ENTRY('USER1.TEST.CNTL') ALL
NONVSAM ------- USER1.TEST.CNTL
IN-CAT --- CATALOG.T70502
HISTORY
DATASET-OWNER-----(NULL) CREATION--------2017.089
RELEASE----------------2 EXPIRATION------0000.000
ACCOUNT-INFO-----------------------------------(NULL)
SMSDATA
STORAGECLASS -----CLASS2 MANAGEMENTCLASS---(NULL)
DATACLASS --------(NULL) LBACKUP ---0000.000.0000
ENCRYPTIONDATA
DATA SET ENCRYPTION-----(NO)
VOLUMES
VOLSER------------T70502 DEVTYPE------X'3010200F' FSEQN------------------0
ASSOCIATIONS--------(NULL)
ATTRIBUTES
IDCAMS SYSTEM SERVICES TIME: 22:45:34 02/26/19 PAGE 2
THE NUMBER OF ENTRIES PROCESSED WAS:
AIX -------------------0
ALIAS -----------------0
CLUSTER ---------------0
DATA ------------------0
GDG -------------------0
INDEX -----------------0
NONVSAM ---------------1
PAGESPACE -------------0
PATH ------------------0
SPACE -----------------0
USERCATALOG -----------0
TAPELIBRARY -----------0
TAPEVOLUME ------------0
TOTAL -----------------1
THE NUMBER OF PROTECTED ENTRIES SUPPRESSED WAS 0
IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0
IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 0
Here is a reference to the SuperC utility
Another approach, if you are more familiar with the Unix side of the house, would be to consider doing these things via USS or TSO. For USS (Unix System Services), check out my blog:
https://makingdeveloperslivesbetter.wordpress.com/2019/01/06/mvs-utilities-an-almost-real-world-example/
If nothing else, reading the code for the various utilities (they are mostly just shell scripts) will help you map from Unix concepts to z/OS (MVS) concepts.
So I'm trying to execute a hive query using the subprocess module, and save the output into a file data.txt as well as the logs (into log.txt), but I seem to be having a bit of trouble. I've look at this gist as well as this SO question, but neither seem to give me what I need.
Here's what I'm running:
import subprocess
query = "select user, sum(revenue) as revenue from my_table where user = 'dave' group by user;"
outfile = "data.txt"
logfile = "log.txt"
log_buff = open("log.txt", "a")
data_buff = open("data.txt", "w")
# note - "hive -e [query]" would normally just print all the results
# to the console after finishing
proc = subprocess.run(["hive" , "-e" '"{}"'.format(query)],
stdin=subprocess.PIPE,
stdout=data_buff,
stderr=log_buff,
shell=True)
log_buff.close()
data_buff.close()
I've also looked into this SO question regarding subprocess.run() vs subprocess.Popen, and I believe I want .run() because I'd like the process to block until finished.
The final output should be a file data.txt with the tab-delimited results of the query, and log.txt with all of the logging produced by the hive job. Any help would be wonderful.
Update:
With the above way of doing things I'm currently getting the following output:
log.txt
[ralston#tpsci-gw01-vm tmp]$ cat log.txt
Java HotSpot(TM) 64-Bit Server VM warning: Using the ParNew young collector with the Serial old collector is deprecated and will likely be removed in a future release
Java HotSpot(TM) 64-Bit Server VM warning: Using the ParNew young collector with the Serial old collector is deprecated and will likely be removed in a future release
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/y/share/hadoop-2.8.3.0.1802131730/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/y/libexec/tez/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Logging initialized using configuration in file:/home/y/libexec/hive/conf/hive-log4j.properties
data.txt
[ralston#tpsci-gw01-vm tmp]$ cat data.txt
hive> [ralston#tpsci-gw01-vm tmp]$
And I can verify the java/hive process did run:
[ralston#tpsci-gw01-vm tmp]$ ps -u ralston
PID TTY TIME CMD
14096 pts/0 00:00:00 hive
14141 pts/0 00:00:07 java
14259 pts/0 00:00:00 ps
16275 ? 00:00:00 sshd
16276 pts/0 00:00:00 bash
But it looks like it's not finishing and not logging everything that I'd like.
So I managed to get this working with the following setup:
import subprocess
query = "select user, sum(revenue) as revenue from my_table where user = 'dave' group by user;"
outfile = "data.txt"
logfile = "log.txt"
log_buff = open("log.txt", "a")
data_buff = open("data.txt", "w")
# Remove shell=True from proc, and add "> outfile.txt" to the command
proc = subprocess.Popen(["hive" , "-e", '"{}"'.format(query), ">", "{}".format(outfile)],
stdin=subprocess.PIPE,
stdout=data_buff,
stderr=log_buff)
# keep track of job runtime and set limit
start, elapsed, finished, limit = time.time(), 0, False, 60
while not finished:
try:
outs, errs = proc.communicate(timeout=10)
print("job finished")
finished = True
except subprocess.TimeoutExpired:
elapsed = abs(time.time() - start) / 60.
if elapsed >= 60:
print("Job took over 60 mins")
break
print("Comm timed out. Continuing")
continue
print("done")
log_buff.close()
data_buff.close()
Which produced the output as needed. I knew about process.communicate() but that previously didn't work. I believe the issue was related to not adding an output file with > ${outfile} to the hive query.
Feel free to add any details. I've never seen anyone have to loop over proc.communicate() so I'm skeptical that I might be doing something wrong.
I am having trouble getting multi-line latex equations to successfully compile when using knitr in R Markdown. When I output as an HTML file it works but when I attempt to output as a PDF I get errors and it will not compile.
The following sample code illustrates the problem. Just change output:html_document to output: pdf_document. You'll see that in HTML output both Test 1 and Test 2 work. But in PDF output only Test 1 successfully compiles.
---
title: "Test"
date: "1/16/2018"
output: html_document
---
## R Markdown
Testing to see if a multi-line latex equation works.
#### Test 1:
$$ f(x) = x^5 + x^3 + x $$
$$ g(x) = y^{x+1} $$
#### Test 2:
\begin{align}
f(x) &= x^5 + x^3 + x \\
g(x) &= y^{x+1}
\end{align}
I am running Mac OS X - High Sierra and R Studio - Version 1.1.383
The following is the error I got:
Rule 'pdflatex': File changes, etc:
Non-existent destination files:
'Untitled.pdf'
------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'pdflatex -halt-on-error -interaction=batchmode -recorder "Untitled.tex"'
------------
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdflatex)
restricted \write18 enabled.
entering extended mode
=== TeX engine is 'pdfTeX'
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
pdflatex: Command for 'pdflatex' gave return code 1
Refer to 'Untitled.log' for details
Latexmk: Use the -f option to force complete processing,
unless error was exceeding maximum runs of latex/pdflatex.
! Paragraph ended before \align was complete.
<to be read again>
\par
l.112
Error: Failed to compile Untitled.tex. See Untitled.log for more info.
Execution halted
I have a requirement wherein I have to submit 10 JCL's. Every JCL is coded to give MAXXCC=0 when completed good.
I want to call all the JCL's from a main JCL so that I don't have to submit all the JCL's manually.
If this is not possible through internal reader, please suggest any other workaround.
THIS IS HOW I CODED THEM CURRENTLY:
//*************************************************************
//* STEP 1: Run job 2
//*************************************************************
//*
//STEP02 EXEC PGM=IEBGENER
//SYSUT1 DD DISP=SHR,DSN=HLQ.MYPDS(JCL2)
//SYSUT2 DD SYSOUT=(,INTRDR)
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//*
//*************************************************************
//* STEP 3: Run job 3
//*************************************************************
//*
//STEP03 EXEC PGM=IEBGENER,COND=(0,EQ,STEP0)
//SYSUT1 DD DISP=SHR,DSN=HLQ.MYPDS(JCL3)
//SYSUT2 DD SYSOUT=(,INTRDR)
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//*
Scheduling System
As other suggested use the scheduling system, I would think all mainframe systems
would have some sort of scheduler. Running Jobs in sequence is there bread and butter
of Scheduling systems.
You do need to learn about scheduling systems and the sooner the better. Ask at work !!!
Other options do exist
JCL Chaining - Each job submit the next job
Basically you can have each job submit the next job
e.g.
Job 1:
// --- Job 1 JCL
//*
// --- JCL to submit Job 2
Job 2:
// --- Job 2 JCL
//*
// --- JCL to submit Job 3
If you use this method I would create a JCL proc (say SUBNEXT)
// --- Job 1 JCL
//*
// EXEC SUBNEXT,NEXT=JOB2
You can use the JCLLIB
statement to use your own PROCLIB (PDS where you store SUBNEXT). To use JCLLIB:
//MYJOB1 JOB ...
//MYLIBS1 JCLLIB ORDER=MY.PROCS.JCL
Jes 3 scheduling
If using Jes-3; There is in-built job control. Only use this
option if you know what you are doing other wise the operation staff will get upset.
Basically make sure you use the flush option
I have a Rexx program that I want to run as a batch job. How can I do this?
This is my program :-
/* Rexx – HELLO – Write Hello World */
Say "hello World"
The program is located as member HELLO in the PDS ME.USER.EXEC.
A valid JOB CARD for my installation is (our environment includes ISPF/PDF as opposed to ROSCOE):-
//MYJOB JOB ,,CLASS=1,MSGCLASS=H,NOTIFY=&SYSUID
Note! this has been written as a tutorial
There are a number of ways that you can run the program via batch. I will cover 3 ways all of which are different according to the environment (i.e. what they can utilise.)
Method 1 - Run the program in a Rexx environment.
This entails running the program IRXJCL and passing the name of the program (i.e. the PDS's member name) via the PARM field (you can also pass parameters; accessing them via a PARSE ARG statement).
IRXJCL requires (normally) 3 DDNAMES they are SYSEXEC (The PDS where the program is located), SYSTSIN (this can reflect terminal input) and SYSTSPRT (this is where terminal output is sent).
Here is the JCL that would work according the information provided above:-
//MYJOB JOB ,,CLASS=1,MSGCLASS=H,NOTIFY=&SYSUID
//*-------------------------------------------------------------------
//RUNPROG EXEC PGM=IRXJCL,PARM=’HELLO’
//*
//* RUN OUR REXX PROGRAM CALLED HELLO
//*
//SYSEXEC DD DSN=ME.USER.EXEC,DISP=SHR
//SYSTSIN DD DUMMY
//SYSTSPRT DD SYSOUT=*
//*-------------------------------------------------------------------
This method, although the simplest (by just a few lines of JCL), is the most restrictive in that it does not allow the use of
TSO/E services, such as TSO/E commands and most of the TSO/E external
functions.
However, as IRXJCL is a Rexx processor, there is no requirement to let
TSO/E know that it is a Rexx program (the first line must include
REXX).
Method 2 - Run the program from a TSO/E environment
This entails running one of the TSO/E batch processing programs IKJEFT01 is used in this example. Alternatives are IKJEFT1A and IKJEFT1B. TSO/E services and commands can be used via this method (e.g. note at end of this method using the TIME command)
Comprehensive information about the differences between the programs
can be found at Writing JCL for command
execution
The JCL for IKJEFT01 is similar to that used in Method 1. An additional DDNAME SYSPROC can be coded. SYSPROC is the DDNAME where CLISTS would be located; You can have Rexx programs found here in addition to SYSEXEC (which isn't required, a suggestion is that both are coded and that SYSEXEC is used for Rexx programs and SYSPROC is used for CLISTS).
It's the requirement that Rexx is on the first line that
differentiates a Rexx program from a CLIST (by the TSO/E processor). Thus, if the Rexx program is found/located via SYSEXEC, if I recall correctly, this negates the requirement).
Another suggestion is to always include REXX in the first line of a Rexx program>
The EXEC statement invokes the IKJEFT01 program instead of IRXJCL. PARM can be used to specify the first command (and therefore our HELLO program). However, as for foreground, you can specify this via the terminal i.e. the SYSTSIN DDNAME.
Here is some JCL that would work for the second method; noting that the HELLO program is invoked via the SYSTSIN DDNAME as instream data :-
//MYJOB JOB ,,CLASS=1,MSGCLASS=H,NOTIFY=&SYSUID
//*-------------------------------------------------------------------
//RUNPROG EXEC PGM=IKJEFT01
//*
//* RUN OUR REXX PROGRAM CALLED HELLO IN A TSO/E ENVIRONMENT
//*
//SYSPROC DD DSN=ME.USER.CLIST,DISP=SHR
//SYSEXEC DD DSN=ME.USER.EXEC,DISP=SHR
//SYSTSIN DD *
HELLO
//SYSTSPRT DD SYSOUT=*
//*-------------------------------------------------------------------
If, for example, the following were used (i.e. added TIME as another
line to SYSTSIN) then the TSO/E TIME command would be run (which would
cause the time to be displayed to SYSTSPRT).
//SYSTSIN DD *
HELLO
TIME
Method 3 - Run the program in an ISPF environment
This method uses the IKJEFT01 program (see method 2 for IKJEFT1A/B alternatives). However, it then uses the ISPSTART command to run the program in an ISPF environment; enabling the use of ISPF services (e.g file tailoring (skeletons) ISPF tables etc).
An ISPF environment has additional requirements in that ISPF libraries need to be allocated in order to start an ISPF environment. At a minimum have the supplied ISPF libraries allocated to ddnames ISPPLIB (ISPF Panels), ISPMLIB (ISPF Messages) & ISPTLIB (ISPF Tables). ISPPROF is where ISPF keeps some profile data for the session, so a temporary store is sufficient (UNIT=SYSDA is often available if not always).
Note you would likely allocate, at a minimum, the installations system
libraries (the TSO/E command LISTA can likely be used to determine
these from a foreground session). Alternately, ask you local friendly system programmers. In the following they are SYS1.ISPPLIB, SYS1.ISPMLIB and SYS1.ISPTLIB.
Here is some JCL that would work for the 3rd method. Note that HELLO is passed as a parameter to the ISPSTART command.
//MYJOB JOB ,,CLASS=1,MSGCLASS=H,NOTIFY=&SYSUID
//*-------------------------------------------------------------------
//RUNPROG EXEC PGM=IKJEFT01
//*
//* RUN OUR REXX PROGRAM HELLO IN A TSO/E/ISPF ENVIRONMENT
//*
//SYSPROC DD DSN=ME.USER.CLIST,DISP=SHR
//SYSEXEC DD DSN=ME.USER.EXEC,DISP=SHR
//ISPPLIB DD DSN=SYS1.ISPPLIB,DISP=SHR
//ISPMLIB DD DSN=SYS1.ISPMLIB,DISP=SHR
//ISPTLIB DD DSN=SYS1.ISPTLIB,DISP=SHR
//ISPPROF DD UNIT=SYSDA,SPACE=(CYL,(10,1)),
// RECFM=FB,LRECL=80,BLKSIZE=0
//SYSTSIN DD *
ISPSTART CMD(HELLO)
//SYSTSPRT DD SYSOUT=*
//*-------------------------------------------------------------------
Note this is not a fully comprehensive, it is an overview that should
suffice for getting started with running Rexx programs in batch.
Additional comments to the above answer.. The below technote may be helpful if you wish to run your REXX exec using ISPF services...
http://www.ibm.com/support/docview.wss?uid=swg21023990
Make sure that the ISPPROF file is concatenated as the first file in ISPTLIB. The example uses a temporary file that will be unique to the job. If the REXX exec does table services you may need an ISPTABL DD. I would suggest using the same file for ISPPROF and ISPTABL and concatenate it first in ISPTLIB. This can be a permanent file if the table needs to be save, however it should not be in use by other jobs or TSO users so as to avoid enqueue errors.
As the previous update stated his answers suffice for the simple REXX exec.
Yet more additional comments to the answer by MikeT...
Suppose you want your JCL to be self-contained: you want to include a REXX exec as an in-stream data set in the JCL, rather than referring to a REXX exec stored in a PDS member.
Here's the "trick" (for IRXJCL; I haven't tested this trick with other programs): specify a single null (X'00') byte as the value of the PARM attribute of the EXEC statement.
To specify the null byte, use the z/OS ISPF editor with HEX ON:
//REXX EXEC PGM=IRXJCL,PARM=' '
66DCEE44444CECC4DCD7CDEDCD6DCDD7707
1195770000057530774E997133B7194ED0D
For example, if you entered PARM=' ' with a space as the value, overtype the 4 of the hex value of that space with a 0.
Here's an example job step containing an in-stream REXX exec that processes the output from a previous step in the same job:
//* PARM value is a single X'00' byte
//REXX EXEC PGM=IRXJCL,PARM=' '
//SYSEXEC DD DATA,DLM=$$
/*
Transposes first line of input CSV into one record per field
Reads CSV from ddname SYSTSIN.
Writes output to ddname SYSTSPRT.
*/
columnSeparator = ","
/* Get the header row */
parse pull row
/* Get column names */
do i = 1 until row = ""
parse value row with columnName "," row
say columnName
end
exit 0
$$
//SYSTSIN DD DSN=&&CSV,DISP=OLD
//SYSTSPRT DD SYSOUT=*
Notes:
In this context, the first line of the REXX exec does not need to contain the string "REXX"
DLM=$$ enables you to use REXX comment syntax (/*) without prematurely ending the in-stream data set
&&CSV refers to a CSV file created by a previous job step (not shown)
I use this "in-stream REXX" technique mostly for ad hoc execs to transform the output of a batch program into what I really want. This technique can be useful for demonstrations and quickly bouncing ideas around; other developers can view the REXX code in situ in the JCL in SDSF output, tweak it, and then submit the tweaked version.