April 1992
Dataflex and TTS
by Michael Steffano
This article concerns implementing Novell's Transaction Tracking System (TTS) in Dataflex
by using the Data Access "C" Source & Library (CS&L). TTS is supported under all current
versions of Netware (except Netware Lite) and comes bundled with the Novell operating system.
BACKGROUND
As of this writing, Novell's Netware accounts for over 65% of all network operating systems (NOS)
sold and it has the largest installed base of any NOS. TTS was initially shipped as an additional
enhancement to Advanced Netware in 1985. As of 1987, with the release of SFT Netware, TTS
became a standard Netware feature. If you are still running a version of Netware without TTS it is
high time to upgrade! Check with Novell on upgrade costs; sometimes it is cheaper to buy Netware
new instead.
A transaction is defined as a set of one or more operations that must be completed together to
maintain file and database integrity. All of the files Netware maintains internally are marked
transactional. Even if you have no other reason you should use TTS to insure Netware's
bindery files retain their integrity! Novell's operating system has the ability to perform transaction
tracking on user designated files also. These must be explicitly marked transactional before Netware
will track them. The following command will flag the file VENDOR.DAT as such:
FLAG VENDOR.DAT T
Once TTS is enabled at the server the operating system will automatically log all changes made
to a file or files from the beginning of the designated transaction until the end of said transaction.
If an interruption occurs on the workstation the file or files that have been changed will be
automatically "rolled back" to the pre-transaction state. If an interruption occurs on the file server
the same process will be automatically performed when the server boots back up. You may also
programmatically back out a transaction by aborting from it within your program. For a technical
explanation of this process you should refer to the Netware Bible- The Netware System Interface
Technical Overview.
WHY USE TTS?
Data can become corrupted in several ways on a network: power loss to a workstation or the server
during transmission, memory or parity errors, software or hardware hangs, network component
failure (such as a repeater failure) and many others. This is a big issue given that DataFlex is
not (yet) a client/server DBMS. Obviously, in an environment where data integrity is of high concern,
or where you may be experiencing database corruption due to hardware problems, TTS holds a high
appeal.
When you add a record in DataFlex 2.3b the following process takes place internally: DataFlex
updates all the indexes in index order (3.0 only updates the affected indexes) followed by the .DAT
header being updated and finally the new data being written to the record. To see the benefits
of TTS take the following example:
you add a record to a ten index file from your workstation,
the first four indexes get written,
the server goes down,
the server comes up,
the indexes are now out of synch,
your boss asks you to work late to oversee the four hour reindex!
As an aside, did you know that DataFlex will update an index if you MOVE data to a field that is
part of an index without a SAVE command? It is very important to realize that index updates
occur when the data is moved to the record buffer, not when the SAVE command is executed.
See how database corruption can happen?
You might be tempted, after reading Novell's literature on TTS, to flag your database and index
files with the "Transactional" attribute. Bad idea! Even assuming that you have already enabled
TTS at the server, Netware will not recognize the index file updates as being part of the transaction
occuring on the database file. Because of the non-standard way DataFlex implements its file locking
mechanism, Netware has no way of "explicitly" knowing when the transaction started. However,
TTS does attempt to recognize "implicit" transactions, so it will initiate transaction logging on the
.DAT file only! Not good ... Actually, this is worse than not using TTS at all.
Unfortunately, our beloved DataFlex does not have the basic transaction commands required to
hook into Netware's TTS (supposedly the DBE to be released "anytime now" will contain built-in
transactional capabilities). These commands, TRANSACTION BEGIN, END, and ABORT tell TTS
when to begin, end and rollback a transaction.
BEGIN TRANSACTION
Tells Netware that the transaction has begun. From this point forward, until the transaction is
completed, aborted or rolled back, all changes to transactional files will be performed as discussed
above.
END TRANSACTION
Tells Netware the transaction has ended.
ABORT TRANSACTION
Tells Netware to abort the transaction.
GET TRANSACTION STATUS
Believe it or not, just because an END TRANSACTION has been issued, it does not mean it has been
written to disk. If you need one hundred percent assuredness then the END TRANSACTION
command must loop until Netware returns a completion code indicating that yes, indeed,
the transaction has been written to disk.
VARIOUS APPROACHES
Given that you know how to program in 'C', the actual Netware service calls to the TTS are very
straightforward. You might be tempted to write some quick 'C' programs and use RUNPROGRAM
WAIT to make the calls, but the overhead of shelling out would lead to some abysmal performance.
Anyway, a much more elegant solution presents itself through the use of Data Access' "C" Source &
Library and FMAC. To use the CS&L you must know how to program in "C", be a licensed DataFlex
developer, and understand the Netware API thoroughly. There are also methods of using
the undocumented OS$CALL command with a Terminate and Stay Resident (TSR) initiating the calls.
Adding new 'C' commands to DataFlex is well documented in the Cs&L, and as I mentioned above,
the calls for TTS services are short and straightforward. The difficult part comes in adding and
modiying your FMAC. You could write brand new commands (like TTS_BEGIN, TTS_END) that
would be explicitly coded into your programs or you could modify the existing LOCK/REREAD/UNLOCK
commands and add the custom 'C' calls to them. This way you could simply rePACK FLEX.CFL and
recompile your programs. All the TTS calls would be built-in automatically. If you have already
invested a lot of time and effort into an application your time would be better spent modifying
existing FMAC commands. If you are writing a system from scratch then you might want to be
able to explicitly invoke TTS services outside of the LOCK/REREAD/UNLOCK commands. And,
there is nothing to stop you from mixing these two approaches.
OS$CALL can be used to communicate with a TSR to perform transaction tracking. This command,
a holdover from the C/PM days, allows the program access to a limited number of system registers
via the interrupt services. Using a TSR to perform TTS services requires two steps. First, you must
know how to write a TSR. This article is not going to delve into that! Second, in a similar fashion to
embedding 'C' calls in your FMAC commands, you would add OS$CALL statements to the same routines.
Then, repack and recompile, and voila! you have TTS services without a modified run-time. However,
now you have to worry about loading the TSR before DataFlex, and, unless you use advanced memory
techniques, your conventional memory size will be slightly smaller.
One of the few drawbacks to using a modified DFRUN.EXE is that the debugging products on the
market will not recognize your FMAC calls to your new 'C' routines. I discussed this problem with
Richard Haendel (developer of the Data Access Debugger) and he told me that the Data Access Debugger
is written in such a way that it also could be licensed (in a similar fashion as the CS&L) so that third
party developers could add their own commands to it. While this is not a perfect solution it would
be a large step in the right direction. As to whether DAC will do this, who knows?
If all of the above is too much for you there are a few third party products on the market that attempt
to address some of the network deficiencies within DataFlex.
Vinga System AB uses the TSR approach with their Vinga DF/TTS solution. You make some minor
modifications in FMAC to the LOCK, REREAD, UNLOCK, ENTER, SAVE, SAVERECORD, and DELETE
commands then recompile your programs. You must load their TSR (VSTTS.COM) before loading
DataFlex and you must make sure your LOCK/UNLOCK pairings are coded correctly. Flag the databases
you want to track TRANSACTIONAL (you might as well do all of them) and you have added an entire
new level of data integrity to your system. One of the advantages to this approach is that your
DFRUN.EXE has not been modified so debugging products like the Data Access Debugger will still
work on all your commands.
Another approach to this issue is to use a different database manager. FLEXQuarters offers a
modified DFRUN called FLEXtrieve that allows you to tap into the Btrieve Record Manager that
comes bundled with Netware. Btrieve is a B+ ISAM file manager that comes bundled with Netware.
This file manager has built in transactional capabilities as well as a host of other features (such
as being server based and SQL compatible). Obviously there are a lot of other points to consider
before going this route as there are some limitations with using Btrieve (supposedly most of
these will be addressed when the new version of Btrieve is released by Novell). FLEXquarter's
implementation is very well done, with thorough documentation and good support. Call
FLEXQuarters for more information.
Some Final Thoughts
* A file flagged transactional cannot be deleted or renamed.
* Also, a file cannot have its attributes changed while it is open.
* Beware of using the Non-Shareable file attribute with DataFlex files. Unless you have
CACHE_BUFFERS=0 in your SHELL.CFG at the workstation, Netware will attempt to
download some of the index files into local RAM. DataFlex ignores this attribute on data files.
* There is a slight performance penalty when using TTS (the operating system has to do an extra
write). However, depending on your server configuration you can minimize or even eliminate
this problem. Your results will probably vary.
Michael D. Steffano has been programming in DataFlex, and working with networks and Netware
since 1982.
Vinga System AB
Gotabergsgatan 28
S-411 34 Goteborg
Sweden
ph +46-31-819850
fax +46-31-812139
FLEXquarters
1125 West Baseline Suite 2-F
Mesa, Arizona 85210 USA
Netware, Netware Lite,TTS, and Btrieve are registered trademarks of Novell, Inc.
FLEXtrieve is a registered trademark of FLEXquarters and Software Services Group.