Dataflex and TTS
by Michael Steffano
This article concerns implementing Novell's Transaction Tracking System (TTS) in Dataflex by using the Data Access "C" Source & Library (CS&L). TTS is supported under all current versions of Netware (except Netware Lite) and comes bundled with the Novell operating system. BACKGROUND As of this writing, Novell's Netware accounts for over 65% of all network operating systems (NOS) sold and it has the largest installed base of any NOS. TTS was initially shipped as an additional enhancement to Advanced Netware in 1985. As of 1987, with the release of SFT Netware, TTS became a standard Netware feature. If you are still running a version of Netware without TTS it is high time to upgrade! Check with Novell on upgrade costs; sometimes it is cheaper to buy Netware new instead. A transaction is defined as a set of one or more operations that must be completed together to maintain file and database integrity. All of the files Netware maintains internally are marked transactional. Even if you have no other reason you should use TTS to insure Netware's bindery files retain their integrity! Novell's operating system has the ability to perform transaction tracking on user designated files also. These must be explicitly marked transactional before Netware will track them. The following command will flag the file VENDOR.DAT as such: FLAG VENDOR.DAT T Once TTS is enabled at the server the operating system will automatically log all changes made to a file or files from the beginning of the designated transaction until the end of said transaction. If an interruption occurs on the workstation the file or files that have been changed will be automatically "rolled back" to the pre-transaction state. If an interruption occurs on the file server the same process will be automatically performed when the server boots back up. You may also programmatically back out a transaction by aborting from it within your program. For a technical explanation of this process you should refer to the Netware Bible- The Netware System Interface Technical Overview. WHY USE TTS? Data can become corrupted in several ways on a network: power loss to a workstation or the server during transmission, memory or parity errors, software or hardware hangs, network component failure (such as a repeater failure) and many others. This is a big issue given that DataFlex is not (yet) a client/server DBMS. Obviously, in an environment where data integrity is of high concern, or where you may be experiencing database corruption due to hardware problems, TTS holds a high appeal. When you add a record in DataFlex 2.3b the following process takes place internally: DataFlex updates all the indexes in index order (3.0 only updates the affected indexes) followed by the .DAT header being updated and finally the new data being written to the record. To see the benefits of TTS take the following example: you add a record to a ten index file from your workstation, the first four indexes get written, the server goes down, the server comes up, the indexes are now out of synch, your boss asks you to work late to oversee the four hour reindex! As an aside, did you know that DataFlex will update an index if you MOVE data to a field that is part of an index without a SAVE command? It is very important to realize that index updates occur when the data is moved to the record buffer, not when the SAVE command is executed. See how database corruption can happen? You might be tempted, after reading Novell's literature on TTS, to flag your database and index files with the "Transactional" attribute. Bad idea! Even assuming that you have already enabled TTS at the server, Netware will not recognize the index file updates as being part of the transaction occuring on the database file. Because of the non-standard way DataFlex implements its file locking mechanism, Netware has no way of "explicitly" knowing when the transaction started. However, TTS does attempt to recognize "implicit" transactions, so it will initiate transaction logging on the .DAT file only! Not good ... Actually, this is worse than not using TTS at all. Unfortunately, our beloved DataFlex does not have the basic transaction commands required to hook into Netware's TTS (supposedly the DBE to be released "anytime now" will contain built-in transactional capabilities). These commands, TRANSACTION BEGIN, END, and ABORT tell TTS when to begin, end and rollback a transaction. BEGIN TRANSACTION Tells Netware that the transaction has begun. From this point forward, until the transaction is completed, aborted or rolled back, all changes to transactional files will be performed as discussed above. END TRANSACTION Tells Netware the transaction has ended. ABORT TRANSACTION Tells Netware to abort the transaction. GET TRANSACTION STATUS Believe it or not, just because an END TRANSACTION has been issued, it does not mean it has been written to disk. If you need one hundred percent assuredness then the END TRANSACTION command must loop until Netware returns a completion code indicating that yes, indeed, the transaction has been written to disk. VARIOUS APPROACHES Given that you know how to program in 'C', the actual Netware service calls to the TTS are very straightforward. You might be tempted to write some quick 'C' programs and use RUNPROGRAM WAIT to make the calls, but the overhead of shelling out would lead to some abysmal performance. Anyway, a much more elegant solution presents itself through the use of Data Access' "C" Source & Library and FMAC. To use the CS&L you must know how to program in "C", be a licensed DataFlex developer, and understand the Netware API thoroughly. There are also methods of using the undocumented OS$CALL command with a Terminate and Stay Resident (TSR) initiating the calls. Adding new 'C' commands to DataFlex is well documented in the Cs&L, and as I mentioned above, the calls for TTS services are short and straightforward. The difficult part comes in adding and modiying your FMAC. You could write brand new commands (like TTS_BEGIN, TTS_END) that would be explicitly coded into your programs or you could modify the existing LOCK/REREAD/UNLOCK commands and add the custom 'C' calls to them. This way you could simply rePACK FLEX.CFL and recompile your programs. All the TTS calls would be built-in automatically. If you have already invested a lot of time and effort into an application your time would be better spent modifying existing FMAC commands. If you are writing a system from scratch then you might want to be able to explicitly invoke TTS services outside of the LOCK/REREAD/UNLOCK commands. And, there is nothing to stop you from mixing these two approaches. OS$CALL can be used to communicate with a TSR to perform transaction tracking. This command, a holdover from the C/PM days, allows the program access to a limited number of system registers via the interrupt services. Using a TSR to perform TTS services requires two steps. First, you must know how to write a TSR. This article is not going to delve into that! Second, in a similar fashion to embedding 'C' calls in your FMAC commands, you would add OS$CALL statements to the same routines. Then, repack and recompile, and voila! you have TTS services without a modified run-time. However, now you have to worry about loading the TSR before DataFlex, and, unless you use advanced memory techniques, your conventional memory size will be slightly smaller. One of the few drawbacks to using a modified DFRUN.EXE is that the debugging products on the market will not recognize your FMAC calls to your new 'C' routines. I discussed this problem with Richard Haendel (developer of the Data Access Debugger) and he told me that the Data Access Debugger is written in such a way that it also could be licensed (in a similar fashion as the CS&L) so that third party developers could add their own commands to it. While this is not a perfect solution it would be a large step in the right direction. As to whether DAC will do this, who knows? If all of the above is too much for you there are a few third party products on the market that attempt to address some of the network deficiencies within DataFlex. Vinga System AB uses the TSR approach with their Vinga DF/TTS solution. You make some minor modifications in FMAC to the LOCK, REREAD, UNLOCK, ENTER, SAVE, SAVERECORD, and DELETE commands then recompile your programs. You must load their TSR (VSTTS.COM) before loading DataFlex and you must make sure your LOCK/UNLOCK pairings are coded correctly. Flag the databases you want to track TRANSACTIONAL (you might as well do all of them) and you have added an entire new level of data integrity to your system. One of the advantages to this approach is that your DFRUN.EXE has not been modified so debugging products like the Data Access Debugger will still work on all your commands. Another approach to this issue is to use a different database manager. FLEXQuarters offers a modified DFRUN called FLEXtrieve that allows you to tap into the Btrieve Record Manager that comes bundled with Netware. Btrieve is a B+ ISAM file manager that comes bundled with Netware. This file manager has built in transactional capabilities as well as a host of other features (such as being server based and SQL compatible). Obviously there are a lot of other points to consider before going this route as there are some limitations with using Btrieve (supposedly most of these will be addressed when the new version of Btrieve is released by Novell). FLEXquarter's implementation is very well done, with thorough documentation and good support. Call FLEXQuarters for more information. Some Final Thoughts * A file flagged transactional cannot be deleted or renamed. * Also, a file cannot have its attributes changed while it is open. * Beware of using the Non-Shareable file attribute with DataFlex files. Unless you have CACHE_BUFFERS=0 in your SHELL.CFG at the workstation, Netware will attempt to download some of the index files into local RAM. DataFlex ignores this attribute on data files. * There is a slight performance penalty when using TTS (the operating system has to do an extra write). However, depending on your server configuration you can minimize or even eliminate this problem. Your results will probably vary. Michael D. Steffano has been programming in DataFlex, and working with networks and Netware since 1982. Vinga System AB Gotabergsgatan 28 S-411 34 Goteborg Sweden ph +46-31-819850 fax +46-31-812139 FLEXquarters 1125 West Baseline Suite 2-F Mesa, Arizona 85210 USA Netware, Netware Lite,TTS, and Btrieve are registered trademarks of Novell, Inc. FLEXtrieve is a registered trademark of FLEXquarters and Software Services Group.