Article – Dataflex and TTS

April 1992

Dataflex and TTS

by Michael Steffano

This article concerns implementing Novell's Transaction Tracking System (TTS)  in Dataflex 
by using the Data Access "C" Source & Library (CS&L).  TTS is supported under all current 
versions of Netware (except Netware Lite) and comes bundled with the Novell operating system. 

BACKGROUND
As of this writing, Novell's Netware accounts for over 65% of all network operating systems (NOS) 
sold and it has the largest installed base of any NOS.  TTS was initially shipped as an additional 
enhancement to Advanced Netware in 1985. As of 1987, with the release of SFT Netware, TTS
became a standard Netware feature.  If you are still running a version of Netware without TTS it is 
high time to upgrade!  Check with Novell on upgrade costs; sometimes it is cheaper to buy Netware
new instead.

A transaction is defined as a set of one or more operations that must be completed together to 
maintain file and database integrity.  All of the files Netware maintains internally are marked 
transactional.  Even if  you have no other reason you should use TTS  to insure Netware's 
bindery files retain their integrity!   Novell's operating system has the ability to perform transaction
tracking on user designated files also.  These must be explicitly marked transactional before Netware 
will track them.  The following command will flag the file VENDOR.DAT as such:

FLAG VENDOR.DAT T 

Once TTS is enabled at the server the operating system will automatically log all changes made
to a file or files from the beginning of  the designated transaction until the end of said transaction.
If an interruption occurs on the workstation the file or files that have been changed will be 
automatically "rolled back" to the pre-transaction state.  If an interruption occurs on the file server
the same process will be automatically performed when the server boots back up.  You may also
programmatically back out a transaction by aborting from it within your program.   For a technical 
explanation of this process you should refer to the Netware Bible- The Netware System Interface 
Technical Overview.

WHY USE TTS?
Data can become corrupted in several ways on a network: power loss to a workstation or the server
during transmission, memory or parity errors, software or hardware hangs, network component 
failure (such as a repeater failure) and many others.  This is a big issue given that DataFlex is 
not (yet) a client/server DBMS.  Obviously, in an environment where data integrity is of high concern, 
or where you may be experiencing database corruption due to hardware problems, TTS holds a high
appeal.

When you add a record in DataFlex 2.3b the following process takes place internally:  DataFlex 
updates all the indexes in index order (3.0 only updates the affected indexes) followed by the .DAT 
header being updated and finally the new data being written to the record.  To see the benefits
of TTS take the following example:

         you add a record to a ten index file from your workstation,
         the first four indexes get written,
         the server goes down,
         the server comes up,
         the indexes are now out of synch,
         your boss asks you to work late to oversee the four hour reindex!

As an aside, did you know that DataFlex will update an index if you MOVE data to a field that is
part of an index without a SAVE command?  It is very important to realize that index updates 
occur when the data is moved to the record buffer, not when the SAVE command is executed.
See how database corruption can happen?
         
You might be tempted, after reading Novell's literature on TTS, to flag your database and index
files with the "Transactional" attribute.  Bad idea!  Even assuming that you have already enabled
TTS at the server, Netware will not recognize the index file updates as being part of the transaction
occuring on the database file.  Because of the non-standard way DataFlex implements its file locking 
mechanism, Netware has no way of "explicitly" knowing when the transaction started.  However, 
TTS does attempt to recognize "implicit" transactions, so it will initiate transaction logging on the 
.DAT file only!  Not good ...  Actually, this is worse than not using TTS at all.

Unfortunately, our beloved DataFlex does not have the basic transaction commands required to
hook into Netware's TTS (supposedly the DBE to be released "anytime now" will contain built-in 
transactional capabilities).  These commands, TRANSACTION BEGIN, END, and ABORT tell TTS 
when to begin, end and rollback a transaction.  

BEGIN TRANSACTION
Tells Netware that the transaction has begun.  From this point forward, until the transaction is 
completed, aborted or rolled back, all changes to transactional files will be performed as discussed 
above.

END TRANSACTION
Tells Netware the transaction has ended.

ABORT TRANSACTION
Tells Netware to abort the transaction.

GET TRANSACTION STATUS
Believe it or not, just because an END TRANSACTION has been issued, it does not mean it has been
written to disk.  If you need one hundred percent assuredness then the END TRANSACTION 
command must loop until Netware returns a completion code indicating that  yes, indeed, 
the transaction has been written to disk.

VARIOUS APPROACHES
Given that you know how to program in 'C', the actual Netware service calls to the TTS are very
straightforward.  You might be tempted to write some quick 'C' programs and use RUNPROGRAM 
WAIT to make the calls, but the overhead of shelling out would lead to some abysmal performance.
Anyway, a much more elegant solution presents itself through the use of Data Access'  "C" Source & 
Library and FMAC.  To use the CS&L you must know how to program in "C", be a licensed DataFlex 
developer,  and understand the Netware API thoroughly.  There are also methods of using 
the undocumented OS$CALL  command with a Terminate and Stay Resident (TSR) initiating the calls.

Adding new 'C' commands to DataFlex is well documented in the Cs&L, and as I mentioned above, 
the calls for TTS services are short and straightforward.  The difficult part comes in adding and 
modiying your FMAC.  You could write brand new commands (like TTS_BEGIN, TTS_END) that 
would be explicitly coded into your programs or you could modify the existing LOCK/REREAD/UNLOCK
commands and add the custom 'C' calls to them.  This way you could simply rePACK  FLEX.CFL and
recompile your programs.  All the TTS calls would be built-in automatically.  If you have already 
invested a lot of time and effort into an application your time would be better spent modifying 
existing FMAC commands.  If you are writing a system from scratch then you might want to  be
able to explicitly invoke TTS services outside of the LOCK/REREAD/UNLOCK commands.  And, 
there is nothing to stop you from mixing these two approaches.

OS$CALL can be used to communicate with a TSR to perform transaction tracking.  This command,
a holdover from the C/PM  days, allows the program access to a limited number of system registers
via the interrupt services.  Using a TSR to perform TTS services requires two steps.  First, you must
know how to write a TSR.  This article is not going to delve into that!  Second, in a similar fashion to
embedding 'C' calls in your FMAC commands, you would add OS$CALL statements to the same routines. 
Then, repack and recompile, and voila! you have TTS services without a modified run-time.  However,
now you have to worry about loading the TSR before DataFlex, and, unless you use advanced memory
techniques, your conventional memory size will be slightly smaller.

One of the few drawbacks to using a modified DFRUN.EXE is that the debugging products on the 
market will not recognize your FMAC calls to your new 'C' routines.  I discussed this problem with 
Richard Haendel (developer of the Data Access Debugger) and he told me that the Data Access Debugger
is written in such a way that it also could be licensed (in a similar fashion as the CS&L) so that third 
party developers could add their own commands to it.  While this is not a perfect solution it would
be a large step in the right direction.  As to whether DAC will do this, who knows?

If all of the above is too much for you there are a few third party products on the market that attempt
to address some of the network deficiencies within DataFlex.

Vinga System AB uses the TSR approach  with their Vinga DF/TTS solution.  You make some minor 
modifications in FMAC to the LOCK, REREAD, UNLOCK, ENTER, SAVE, SAVERECORD, and DELETE 
commands then recompile your programs.  You must load their TSR (VSTTS.COM) before loading
DataFlex and you must make sure your LOCK/UNLOCK pairings are coded correctly.  Flag the databases
you want to track TRANSACTIONAL (you might as well do all of them) and you have added an entire
new level of data integrity to your system.  One of the advantages to this approach is that your 
DFRUN.EXE has not been modified so debugging products like the Data Access Debugger will still 
work on all your commands.

Another approach to this issue is to use a different database manager.  FLEXQuarters offers a 
modified DFRUN called FLEXtrieve that allows you to tap into the Btrieve Record Manager that 
comes bundled with Netware.  Btrieve is a B+ ISAM file manager that comes bundled with Netware.
This file manager has built in transactional capabilities as well as a host of other features (such 
as being server based and SQL compatible).  Obviously there are a lot of other points to consider
before going this route as there are some limitations with using Btrieve (supposedly most of 
these will be addressed when the new version of Btrieve is released by Novell).  FLEXquarter's 
implementation is very well done, with thorough documentation and good support. Call 
FLEXQuarters for more information.

Some Final Thoughts
* A file flagged transactional cannot be deleted or renamed.
* Also, a file cannot have its attributes changed while it is open.
* Beware of using the Non-Shareable file attribute with DataFlex files.  Unless you have 
  CACHE_BUFFERS=0 in your SHELL.CFG at the workstation, Netware will attempt to 
  download some of the index files into local RAM.  DataFlex ignores this attribute on data files.
* There is a slight performance penalty when using TTS (the operating system has to do an extra 
  write).  However, depending on your server configuration you can minimize or even eliminate 
  this problem.  Your results will probably vary.

Michael D. Steffano has been programming in DataFlex, and working with networks and Netware 
since 1982.  

Vinga System AB
Gotabergsgatan 28
S-411 34 Goteborg
Sweden
ph  +46-31-819850
fax +46-31-812139

FLEXquarters
1125 West Baseline Suite 2-F
Mesa, Arizona 85210 USA

Netware, Netware Lite,TTS, and Btrieve are registered trademarks of Novell, Inc.
FLEXtrieve is a registered trademark of FLEXquarters and Software Services Group.