The Computer Journal, Issue 35
Z-System Corner
© Jay Sage
Reproduced with permission of author and publisher.
My column about shells and WordStar Release 4 (WS4) in TCJ issue #33 prompted more than the usual level of commentary. There were extensive discussions on Z-Node-Central and the Lillipute Z-Node (the official TCJ bulletin board), and several messages reached me over the ARPA network. Not all of the comments were favorable, but I was nevertheless happy to receive them. They helped further clarify my thinking on the very important subject of shells and have spurred me on to prove my points by actually converting WS4 to a ZCPR2-style shell! After a bit of follow-up discussion, I will describe how this conversion was accomplished.
Corrections
There were some things I said in the previous column that were factually wrong, and before I do anything else I wish to correct them.
First, I stated that the Z-System code in WS4 was written by someone other than MicroPro. I was wrong. David McCord, who was vice president at Echelon at the time WS4 was developed, sent me a message with the facts of this matter. Echelon, through staff like David and through published materials, educated Peter Mireau of MicroPro on the facilities and philosophy of Z-System. Peter did all the actual programming, so the coding mistakes were his fault, not Echelon's or David's.
From a broader perspective, however, as I stated in the previous column, the real culprit was inadequate testing. Bugs in the code would have been discovered and conceptual issues clarified had more people in the Z community been involved as beta testers. There are so many different styles of using Z-System that it takes a number of testers to uncover problems. Within days after copies of WS4 were delivered to users of my Z-Node, I started getting questions about strange behavior exhibited by WS4, behavior that turned out to result from its operation as a shell.
A second mistake in the earlier column was my implication that WS4 does not get its own name from the ZCPR3 external file control block (XFCB). I no longer remember what made me think that this was the case, but David McCord assured me (and I have now verified for myself) that WS4 does, indeed, get its name from the XFCB when it sets up the shell stack entry.
Finally, one reader reported to me that my WSSHLOFF routine (the one that completely disables shells while WS4 is running and reenables them when WS4 terminated) crashed his system. Unfortunately, a large number of misprints crept into the listings in going from my disk file to the printed pages. Most of the typos were obvious, but one was compounded by a double error. In the WSSHLOFF listing, the value for EXITSUB was printed as 03BVh. The 'V' was obviously a mistake, and clever readers looked at the similar listing for WSSHLFIX, where the value was given as 03B3h. This, regrettably, looks correct but was also a typo. The proper value is 03BEh. We hope that Art Carlson will make sure that the listings in this column are transcribed accurately (so that all the mistakes will be mine!).
More WS4 Comments
While on the subject of WS4, I would like to add a few further comments about how it works. Not surprisingly (considering when it was developed), in creating its shell stack entry WS4 does not make use of the facility introduced with ZCPR version 3.3 that allows a program to determine from the XFCB not only its name but also the directory from which it was actually loaded (the user number is at offset 13 and the drive, with A=1, at offset 14).
As a result, in order for WS4 to be reinvoked as a shell, the command search path must include the directory in which WS4 is located. I mention this here as a reminder and suggestion to authors of new or updated shells and error handlers that they use this Z33 facility to avoid the requirement that the program be on the path and to speed up loading of the program (by eliminating any search for it). My WordStar conversion described later adds this feature.
With WS4 it is generally necessary that the command search path include WS4's directory for an additional reason. I learned the hard way that when WS4 runs under Z-System, it pays no attention to the drive and user number that WSCHANGE specified as the location for the overlay files; it only uses the search path to try to locate them.
This is a problem for me because, as I have explained at length in previous columns, I put only my small RAM disk on the path and use ARUNZ aliases to invoke all programs except the very few that fit on the RAM disk. With this approach, there is no way to get WS4 to find its overlay files. The conversion addresses this problem also.
ZCPR2 vs. ZCPR3 Shells
I would not like to take up again one of the subjects raised in issue #33: ZCPR2-style versus ZCPR3-style shells.
First an aside. Shells seem to be a surprisingly emotional issue. I thought my earlier column presented a fairly carefully and calmly reasoned discussion of some aspects of shells, including their pros and their cons. Some readers, however, took great offense at my even questioning the current method of implementing shells or of what some people are trying to do with them.
One reader went so far as to suggest that I had no business commenting on the subject when, by my own admission, there are a number of shells that I have never used. Besides the fact that this is hardly a reasoned argument, I would like to make sure that the following facts about shells are fully appreciated.
ZCPR3-style shells are a facility of the command processor. Without special code in the CPR, there would be no such shells. As the author of the two latest versions of the ZCPR command processor, I think I can speak with some authority (though certainly not with infallibility) on the subject, since in writing that code I had to consider the issue of shells rather carefully from a rigorous theoretical viewpoint.
ZCPR2-style shells, quite the contrary, are not a facility of the command processor; they are a facility of the individual shell programs. Their functioning depends only on the operation of the multiple command line facility. The command processor does not treat a Z2 shell command any differently than it treats any other command. This is really the key to the difference between the two shell implementations.
In the previous column I stated: "...I am coming to the conclusion that only programs like history shells...should be implemented as ZCPR3-style shells. Other programs, like ZFILER and WordStar should use the ZCPR2 style." I then invited readers to enlighten me if I was missing some important point. I got some responses to this invitation, but no one yet has offered me any evidence that I had missed any important point.
One reader reiterated essentially the same difference between Z2 and Z3 shells that I attempted to demonstrate with my example in which WordStar was invoked in a multiple command line. Apparently the point bears repeating.
This reader presented the point using a command line like the following:
ZFILER;ECHO TESTING
Under ZCPR2, ZFILER would run and present its file display to the user. If the user generated a command line "CMDLINE" as the result of a macro or in response to the prompt after the 'Z' command, a Z2-shell version of ZFILER would build the command sequence "CMDLINE;ZFILER" and insert it into the multiple command line buffer just before the next command to be executed. This would give:
CMDLINE;ZFILER;ECHO TESTING
The user's command line would run, and then ZFILER would be invoked again. Only on termination of ZFILER would the last command, "ECHO TESTING", be performed.
A Z3 shell would respond to the same command line from the user in quite a different way. As before, ZFILER would be invoked first. It would determine from the Z3 message buffer that it had been invoked manually and would respond by pushing its own name onto the shell stack. Then it would terminate. The command processor would then proceed to run "ECHO TESTING". Only after that, once the command line was empty, would ZFILER be reloaded, this time as a shell. Recognizing its shell status, it would now display its screen of file names and do its real work.
The reader who submitted this example, if I understood him correctly, viewed the Z3 behavior as correct and the Z2 behavior as wrong. If you are an experienced Z-System user, you will probably recognize in this reader a fellow expert (and, indeed, he is). He is so used to ZCPR3 that he no longer notices that it is the behavior of the Z3 shell that is truly bizarre!
Consider the following two command lines:
(1) ZFILER;ECHO TESTING
(2) ECHO TESTING;ZFILER
We have already analyzed the first one; the second one can safely be left as an exercise for the reader. We will simply state the answer that under ZCPR3 they will accomplish exactly the same thing! This is hardly a result that conforms to intuition, and I still remember in my early days as a Z-Node sysop trying to explain to quite a few users why the second command on a VFILER command line executes first!
Under ZCPR2 the result is just what one would expect. In the first case, ZFILER runs first, and ECHO runs only after the user terminates ZFILER using its 'X' command. In the second case, ECHO runs first and ZFILER second. In other words, with Z2 shells, commands are executed in the order they are entered, a notion that does not require long experience and great expertise to understand and get used to! And it gives the user a greater measure of control.
Mixed Z2 and Z3 Shells
The same reader submitted another interesting example that illustrates the confusing behavior that can arise when Z2 and Z3 shells are mixed. Here we assume that WordStar has been implemented as a Z2 shell and ZFILER as a Z3 shell. Suppose we use the 'R' command of WordStar to enter the command "ZFILER". WS4, as a Z2 shell, would generate the command line
ZFILER;WS
ZFILER, as a Z3 shell, would install itself on the shell stack and proceed to execute "WS". ZFILER would not run in its file maintenance mode until after we terminated WordStar.
This is, admittedly, probably not what one intended, since we most likely entered the ZFILER command with the intention of doing some file maintenance before returning to WordStar. On the other hand, it is certainly no more bizarre than what we saw in our earlier example.
If both WS4 and ZFILER were Z3 shells, then the invocation of ZFILER from the WS4 'R' command would cause it to become the active shell (the one on the top of the shell stack). The WS4 shell would be pushed down in the shell stack, and ZFILER would take control. With a little thought, however, you will see that the same is also true if both ZFILER and WS4 are Z2 shells!
The strange behavior with the mixed shells in the above example arises in part because ZFILER was not really being used as a shell in the Z3 sense, namely, as a replacement for the command processor's command-line input routine. It was intended as a file maintenance utility.
Suppose we had entered the command "EASE" (the Z3-type history shell) instead of "ZFILER" from our Z2 version of WordStar. This would establish EASE as the current shell and return to WordStar. That behavior would not seem strange, because in this case we would be thinking of our EASE command as establishing the shell to be used in place of the command processor the next time the command processor needed a new command line. So long as WordStar is running, there is no need for EASE to do anything. We expect it to take effect only after we are finished using WordStar.
Nested Z2 Shells and Recursive Aliases
Although I had once thought that the Z3 shell stack was required in order to nest shells, I showed in the earlier column that this is not the case. Z2-style shells can, in fact, be nested more flexibly. There is no predetermined limit to the nesting depth or to the amount of information that can be passed with each shell command line. The only limit is imposed by the length of the multiple command line buffer, just as with the nesting of aliases.
With the standard shell stack configuration of 4 32-byte entries, if a shell command uses only 16 bytes, 16 bytes are wasted. On the other hand, if a shell command needs 48 bytes to hold its information, it cannot run at all under this configuration (NZ-COM can come to the rescue by allowing the shell stack configuration to be changed on the fly). With Z2 shells, these problems go away. In 64 bytes of command line, one can have two 32-byte shell commands or a combination of one 16-byte shell command and one 48-byte shell command (or five 12-byte shell commands).
I did overlook one point when I described putting data for the shells on the command line. In the Z3 shell stack, one can include, after the shell command's terminating null, any binary data that one wishes. Thus 256 values are possible for each extra byte in the shell stack entry.
In order to carry shell data on the command line, several additional constraints apply. First, the command processor strips the high bits off all characters in the command line, so only 128 values are available to start with. Secondly, the null character cannot be used because the command processor would interpret this as the end of the command line (that leaves 127 values). Finally, letters are converted to upper case, thereby making the characters from 'a' to 'z' inaccessible (scratch another 26). This leaves only 101 possible values out of the original 256. Moreover, extra characters are required as flags to signal the program to consider itself as having been invoked as a shell (a service provided in ZCPR3 by a flag in the message buffer). All of these things reduce the efficiency with which the space in the command line buffer can be used compared to the space in the shell stack.
One reader pointed out that recursive aliases cannot be used with Z2-type shells. This is true... but only if one is using the pseudo-recursive alias that I invented. This kind of alias accomplishes a crude approximation to recursion by discarding any pending commands in the command line buffer. This will, indeed, discard any shell reinvocation commands. However, if one uses the logically sound and rigorous recursive alias technique invented by Dreas Nielsen (see my column in issue #28), there is no problem. It sometimes pays to do things right!
In fact, it seems to me that the Z2 shell is, in essence, a recursive alias, a program that keeps invoking itself. And this is just what most (if not all) Z3 shells actually do. I am still awaiting an example of something (good) that a Z3 shell can do that cannot be done in some equivalent way with a Z2 shell or recursive alias.
The Real Difference Between Z2 and Z3 Shells
After much reflection, I think I have finally put my finger on the fundamental distinction between Z2 and Z3 shells. It derives from the facts I alluded to earlier: that the Z3 shell is a true creature of the command processor and the Z2 shell is not.
Here is an example that will illustrate the point. Suppose the history shell EASE were implemented as a Z2-style shell and that while it is running, we issue the command "DIR". EASE will insert into the command line a sequence like the following:
DIR;EASE
DIR will run, and then EASE will be reinvoked. Looks fine! But now suppose the user enters the command "IF EXIST FN.FT". EASE will then generate the command line
IF EXIST FN.FT;EASE
If the file FN.FT exists, this will again work just fine, but suppose the file does not exist. Then the system will enter a false flow state, and the EASE command (and perhaps other commands pending in the command line after it) will be flushed by the command processor. The shell function will be lost, and any other pending commands will be processed in an unintended way.
For a Z2 shell to function properly in general, all command lines inserted by it must result in the same flow state at the end of the command line as at the beginning. With a MENU shell it could be possible for the system designer to guarantee this, since he can control which commands are generated by the shell. With a history type shell it would be nearly impossible to ensure that this condition would always be met.
The critical feature of shell processing under ZCPR3 is that _flow processing is suspended during the operation of shells_. This allows them to run, as they must, even after the user has passed a command that leaves the system in a false flow state. The ZCPR33 Users Guide goes into some detail on this matter, and had I remembered better what I wrote there, it would not have taken me this long to come to the essence of the Z2-vs.-Z3 shell issue.
Some users of ZCPR33 have modified the way the command processor deals with flow control in shell processing. No one has yet convinced me of the value of this (the risks are undeniable). It still seems to me that Z2-type shells and recursive aliases can accomplish the same thing, but in a logically sound way.
I have extended an invitation to Dreas Nielsen to write a series of columns for TCJ explaining his very powerful shell programs. Since he is also one of the people who has made this modification to the CPR, perhaps he will also present the other side of this story and explain why it is necessary or desirable to treat shells the way he does.
Remaking WordStar Release 4
When I first received my copy of WordStar 4 and encountered problems with the way it handled shells, I fired up the DSD debugger and tried to figure out how to fix it. After a considerable amount of rummaging about in the code (and especially trying to figure out what was going on inside WS.OVR), I gave up. Later I tried again... and failed again. In the course of preparing this column, I decided to have one more go at it, and this time things started to click.
The patches I will describe here are preliminary and have not yet been extensively tested. In fact, as I write this, I am the only one who has used them, and you know what I said above about the dangers of a test program that does not involve a variety of Z-System users. So, you are hereby recruited, if you are willing, to join the test program.
Since I may very well have made some mistakes, and since there are further changes that people may want to make (let's hear your suggestions), I will not only give the results; I will describe the process by which these patches have been developed.
Cracking the Code
The first step toward changing the code was figuring out how the virgin WordStar was doing what it did. In particular, I wanted to locate routines related to Z-System functions, so the first thing I tried was searching for all references to address 109h, which contains the address of the Z-System environment (ENV). Any WS4 feature that made use of a Z-System facility would have to get information from the ENV.
As best I recall, this did not turn up many references and did not particularly help (though it was a good idea, and that's why I mention it). In the end, I just started tracing the code from the beginning, figuring that WS4 would have to determine fairly early whether it was running under Z-System or standard CP/M. This turned out to be correct, and very soon I came to the key Z routine, at address 0AA4h in WS.COM. This routine returns the address and size of a Z-System module specified by an offset passed in the E register.
Having discovered this routine, I used DSD to find all references to it in WS.COM and WS.OVR. They occur with the following values of E:
E = 9h the command search path (PATH) E = 15h the named directory register (NDR) E = 18h the multiple command line buffer (MCL) E = 1Eh the shell stack (SHL) E = 22h the message buffer (MSG) E = 24h the external file control block (XFCB)
Setting up the Shell Stack
The block of code beginning around address 3CBFh in WS.OVR makes references to MCL, XFCB, and SHL. I guessed correctly that this had to be the code where WS4 sets up its shell stack entry. (This block of code, by the way, is where the shell-pushing mistake occurs for the case where the shell stack is currently empty.)
The patch for this part of WS.OVR (see Listing 1) modifies this code. First of all, since WS4 is going to operate as a Z2-type shell, we do not want it to do anything with the shell stack. It is easy to disable the code by simply skipping over it, but one has to watch out for subtleties. Indeed, in order for the 'R' command to use the MCL and not chain using the greatly inferior CP/M method, WS4 has to think that the shell entry was established successfully.
I noticed that a flag was being set into address 2200h, and I surmised that it is used by WS4 to show that it is running under Z-System. In the patch, I set this flag even though the shell stack entry is not being set up. I have not examined all references to this flag, and there is a chance that there are additional, more complex effects. If any problems appear with the patched version of WordStar, this flag might be involved. For the initial attempt at fixing WS4, I just took the easiest course of action, and so far it appears to have worked.
It seemed foolish to waste space in WS.OVR by doing nothing more than setting the flag and jumping to where the original code resumed (60AAh). Instead, I have used the space to compute the command line necessary to reinvoke WordStar. The code gets not only the name by which WordStar was invoked but also the drive and user number from which it was loaded. A command line of the form ";DUU:WSNAME" is generated.
There is one extra step in this part of the patch. When running as a Z3 shell, WS4 knows from the command status flag in the message buffer when it was invoked as a shell so it can put up the press-any-key message before clearing the screen and resuming operation. As a Z2 shell, WS4 cannot use this facility. Instead, a signal has to be included in the command tail. For reasons that I will not go into in full detail, I chose for this signal a comma at the very end of the tail. Very briefly, the comma is a handy character because it is not parsed into the default file control blocks, where a program could confuse it with a file name.
The final reinvocation command line, with its terminating null, takes the form
;DUU:WSNAME ,<0>
Since I could not be sure that this section of overlay code would persist in memory until the command would be used, I store it at the top of the WS4's user patch area (MORPAT).
The Initialization and Termination Patches
Having made the above modification, we must make two others in order to remain consistent. First, we must modify the WS4 initialization code in which it determines whether or not it was invoked as a shell. This is the patch at address 1A2Fh in WS.COM. The patch calculates the address of the last character passed in the command tail and checks to see if it was a comma. If not, it proceeds with normal operation of the program.
If there is a comma, the shell-wait message must be displayed until the user presses a key. But one must also remove the comma from the command tail to ensure that WordStar not think it has been passed a file name. At present I do this by replacing the comma with a space. This is not rigorous, but it seems to work, since WS4 is apparently not confused by a tail consisting only of spaces (unfortunately, a number of programs are).
Since WS4 no longer pushes its name onto the shell stack, we must also prevent it from popping the shell stack when it terminates. This is the patch at address 13CEh in WS.COM. This is the easiest patch of all, since we simply have to skip some code. As an additional benefit, this frees up about 40 bytes of space that we use for some of our other patch code.
Fixing the 'R' Command
Now we come to the main item in this set of patches - the code that makes the 'R' command work as a ZCPR2-type shell. The new code here is much more complex than what it replaces, and we can only fit part of it at the original location 67B2h in WS.OVR. We put what we can there and continue with the rest in the MORPAT area in WS.COM.
The basic strategy is to take the command line entered by the user in response to the 'R' command prompt, append the WS reinvocation command (including its semicolon separator), and append any remaining command line pending in the multiple command line buffer (if there is one, it will begin with a semicolon also). If there is enough room for the result in the MCL, then it is moved there and chained to. If not, a warning message is displayed on the screen until a key is pressed, and the user command is ignored.
To implement this strategy, I chose the simplest method I could think of. Since the 'R' command operates from the WordStar no-file menu, the entire WS edit buffer is available as a scratch area. I picked an arbitrary address of A000h for a buffer in which to build the new command line. Again, rigorous code calculate the address. My code is a quick-and-dirty solution.
Finding the Overlay Files
As I noted earlier, with my style of operation, WS4 had trouble finding its overlays. To solve that problem, the patch includes an optional section to install an internal search path for the overlay files. This patch is installed at address 0F5Fh in WS.COM, where it replaces a call for the location of the Z-System path with a call to a routine that returns the address and size of an internal path. In Listing 1 the internal path has the single element B4:, the directory in which I keep my WordStar program files. You can put any values you want here.
Installing the Patches
It is not possible to install the patches in WS.OVR using MLOAD or a debugger, because the OVR file is too large to load entirely into memory. ZPATCH, on the other hand, can handle the job splendidly. ZPATCH assumes an offset of 0000 for a file of type OVR, while the addresses in the listing are those shown when the file (as much as can fit) is read into memory under a debugger. To make things consistent, you should use the ZPATCH 'O' command to set the offset to 100.
Key in the new data carefully, checking from time to time that the address is still correct. Also, be careful not to go beyond a record boundary while in ZPATCH. It wraps from the end of the record back to the beginning of that record without warning (this really gave me grief until I caught on to the problem). When you get to the end of the current record, write it out (^W), advance to the next record (>), and reenter edit mode (E). Then you can resume entering data.
The attached listing was made with a specially configured version of the SLR Z80ASM assembler. Normally, I have it display addresses in logical order for easier interpretation. For hand keying of a patch, however, it is far more convenient to have the bytes of a word in physical order. Just watch out when reading the displayed symbol values. They, too, are stored in byte-reversed format.
It is possible to use MLOAD to install just the patches for WS.COM. Simply delete the parts of WSPAT.Z80 that refer to patches in WS.OVR and assemble the remaining code to a HEX file.
Enjoy playing with (and using) this different (improved) version of WordStar 4, and let me know what you think and what further suggestions you have.
[This article was originally published in issue 35 of The Computer Journal, P.O. Box 12, South Plainfield, NJ 07080-0012 and is reproduced with the permission of the author and the publisher. Further reproduction for non-commercial purposes is authorized. This copyright notice must be retained. (c) Copyright 1988, 1991 Socrates Press and respective authors]