Big News in This Release
TOOLForge, the IDE for TOOL, has been enhanced to allow visual creation of XML Forms. This enhancement will save anyone interested in using XMLForms with TOOL the pain of creating form definition files by hand. There is more work to be done here with the more advanced XMLForm control types, but I'm wagering the existing feature set will address most user's needs. However, this program can only be released in non-source form. There is just too much commercial library code in this project to have it any other way. See below for more information on how to work with TOOLForge to debug TOOL scripts.
Introduction
TOOL (Tiny Object Oriented Language) is an easily-embedded, object-oriented, C++-like-language interpreter. The language, and indeed a significant part of the core of the TOOL engine, is based on the BOB project, a work that was originally developed by David Betz covered in previously published issues of Dr. Dobb's Journal.
The original language interpreter was implemented entirely in K and R style C. The TOOL interpreter uses an object-oriented design of the original BOB interpreter, and also includes countless dozens of other enhancements to the original project. The object oriented redesign was accomplished by packaging the functionality of the original project into a small set of inter-operable C++ classes. These classes were then aggregated into a single container class that acts as a facade over the entire system.
One of the primary purposes for the redesign of the original interpreter was to bring the code up to “more modern standards”. However, the key benefit of packaging the interpreter into a single container class is that any application can easily create an "interpreter context" simply by invoking a single class. Applications can also run multiple "stand-alone" interpreter sessions simultaneously simply by having multiple instances of the wrapper-class in scope at once.
The purpose of this article is to introduce the TOOL interpreter and language from the perspective of a person who has a desire to include a scripting solution as part of his project. To this end, the following topics will be discussed:
- The design of the C++ classes that make up the TOOL core.
- How to integrate TOOL into your application.
- How to extend TOOL for your custom requirements.
- How to program in TOOL.
- An examination of the TOOL classes framework, which is a set of classes written in the TOOL language and which provides a set of tested and ready-to-use TOOL classes and that also illustrates the entire set of TOOL intrinsic functions (the TOOL API).
These days, interest in interpreters seems to be at an all time high, and there are many interpreters and technologies to choose from. This being the case, why should anyone consider the TOOL engine as part of a project’s solution? A short list of considerations follows below, which have driven me to include TOOL as part of my own products.
- Lack of Dependencies. When considering the inclusion of a scripting system that is not an integral component of an Operating System installation, in order for your product to run, it is dependant on the correct installation and configuration of a third party product. This kind of dependency is a potential “break point” and therefore is one that I prefer to avoid if at all possible.
- Easily Embedded. Truly embedding a third-party scripting-engine into your solution is not always an easy prospect; especially if part of your goal is to exchange data between your application context and the scripting context (in effect incorporating the interpreter to the degree that it is an integral part of the project, and not just “stuck on the side of it”). Often times, data must be “marshaled between” the two contexts using APIs or syntax which is not always that easily understood or straightforward. So, to address this, one of my goals with TOOL was to make the exchange of data between the two contexts as simple as possible. To this end, I have included an application context object into the design of TOOL. This application context allows for a simple, loosely coupled, yet effective, means of exchanging data between the application environment and the script context.
- Easily Extended. There are certain occasions when part of the reason for embedding a scripting system into a project is to expose specific portions of an application’s functions into the scripted context (in effect making your programs programmable). With most interpreters available today, the prospect of extending them can be a daunting one. In examining the source for some of the more common interpreters available at this time, I found dozens and dozens (in some cases hundreds) of files whose organization is not readily apparent to the casual observer. This being the case, it can be extremely difficult as an “outsider” to understand where and what to edit to extend the functionality of these engines. And if your project is “typical”, it most likely has tight deadlines that will not afford you the chance to take the time required to understand the internal organization of these projects to the point of being able to modify their run-times. And, even if you succeed in that task, now you are potentially the proud owner of a customized, “one-off” version of that distribution that brings its own set of maintenance challenges and may require constant validation of your extension with all future versions of the selected engine. In direct contrast, TOOL remains an easily pliable engine and is very easily extended. The techniques to extend TOOL will be thoroughly discussed in this article.
- No Licensing Hassles. Even so called “open source” interpreter engines can be encumbered with additional license fees and other costs if they are included as part of a commercial product. For “low volume” products, these kinds of costs can be prohibitive. And between you, me, and the water cooler it seems somehow wrong that anyone feels they have the right to charge monies for something that is “supposed to be free” so I tend to shy away from these situations on principle.
The design of the C++ classes that make up the TOOL
As was mentioned, TOOL is comprised of a small number of core classes. The bulk of the files included in this project are extensions to that basic engine. Since there is not enough space to cover all these classes in great depth, I’ll have to leave the extended analysis to the reader, and therefore my goal will be to attempt to impart an understanding of the core classes within the TOOL engine. I hope to give you enough information to allow you to get your bearings in the project; but down in the guts understanding is best gained by wrestling the project to the mat.
Listing 1. Simple example of how to drive the TOOL engine. What we are shown here is a data exchange variable called “oArgs” that will be used to share data between the application and the scripting engine. “oArgs” is then passed into an instance of a VMFacade object which is the container for the entire TOOL scripting engine. All that is needed to run a script then is issue a call to compile and to execute on the facade object. The script can return variables back to the application through the oArgs container.
int main(int argc, char* argv[])
{
SCRIPT_ARGUMENTS oArgs;
oArgs.insert( SCRIPT_ARGUMENTS::value_type( std::string( "TestVector" ) ,
new CAppContextVariant( "On" ) ) );
oArgs.insert( SCRIPT_ARGUMENTS::value_type( std::string( "TestStack" ) ,
new CAppContextVariant( "On" ) ) );
oArgs.insert( SCRIPT_ARGUMENTS::value_type( std::string( "TestQueue" ) ,
new CAppContextVariant( "On" ) ) );
oArgs.insert( SCRIPT_ARGUMENTS::value_type( std::string( "TestMap" ) ,
new CAppContextVariant( "On" ) ) );
oArgs.insert( SCRIPT_ARGUMENTS::value_type( std::string( "TestByteArray" ) ,
new CAppContextVariant( "On" ) ) );
oArgs.insert( SCRIPT_ARGUMENTS::value_type( std::string( "TestString" ) ,
new CAppContextVariant( "On" ) ) );
VMFacade* poScript = new VMFacade();
poScript->SetVar( &oArgs );
poScript->CompileFile( "C:\\Projects\\ScriptEngineTester\\CoreClasses.tool" );
poScript->Execute( "TestAll" );
return( 0 );
}
The first class that the users of TOOL are going to encounter is the VMFacade class (declared in VMFacade.h and implemented in VMFacade.cpp). This class is the single wrapper class that aggregates all components of the entire TOOL system; and it will be the one that you’ll want to instantiate in your projects to get instances of the scripting engine. You can see from the short example in Listing #1 just how simple this class is to use. After you create an instance of this class, you simply need to compile a file (or a text block if you’d prefer) with a call to CompileFile() or CompileText().
If you need to share variables from your program with the script engine, or if you need to retrieve results from the execution of a script, you can easily accomplish this by placing parameters/variables into an instance of a SCRIPT_ARGUMENTS class. This STL derived map is set up to store all the variable types that can be exchanged between the application and the scripting environment. All variables in the application context are stored under a string key, and the value type is a variant type of object that can be one of the following types: String, Long, Boolean, Double, DWORD, DateTime, ByteArrray, and a NULL object type. The CappContextVariant class is declared in the file: VMCoreVirtualMachine.h.
The script can pull values out of the container by name while it is running using a call to the TOOL API function: GetAppEnvField(). The script can also use the context to return variables created in the script’s execution back to the hosting application using the TOOL API function: SetAppEnvField().
In order to facilitate communication between the hosting process and the scripting environment then: simply declare an instance of a SCRIPT_ARGUMENTS class, place variables into it, and then share that context with the VMFacade class through a call to the SetVar() on the VMFacade object. After the script has completed, the hosting application can examine this map for any results sent back to the application from the scripting engine.
With all this setup being accomplished, the next thing to do is to invoke the scripting engine with a call to the Execute() method on the facade. One nice thing about the TOOL engine is that you can specify the entry point for the script execution simply by declaring the entry point in the call to Execute(). This capability can be handy in the case where you may have multiple script programs included in a single script file and you want to “select which one to run” based on logic in the main application. Another way this feature can be used, is to define the “standard interface” for all your scripts and then simply define all scripts so that they have the same entry point(s).
I’ve used both approaches in my projects. The first approach is handy if you would like to have multiple entry points (say for example, a kind of a macro language system available to your project) in a single script file. The second approach is handy if you want to describe a set of standalone script files that all will be invoked by the hosting program in the same fashion as “scripted objects”. In my case, I’ve employed an Init/Run semantic quite successfully wherein all scripts must have Init() and Run() functions in them that do the work specific to that script.
Peeking behind the VMFacade class, the next major class encountered is the VMCore class (declared in VMCore.h and implemented in VMCore.cpp). This class is also a facade class over the rest of the system, but does more of the detailed operations required to stitch the TOOL runtime altogether. As such, this class aggregates all the other major components of the TOOL engine. The VMCore class drives the actual compilation and execution operations of the TOOL system. It also initiates all the other major components in TOOL. As part of constructing the TOOL engine, this class provides all the major components with pointer references to the other major components. Finally, this class also catches all exceptions thrown from the TOOL engine. Exceptions can be thrown during compilation if there are syntax errors detected in the script. Exceptions can also be thrown during script execution if one of the integrity checks performed by the TOOL engine fails. Finally, exceptions can be initiated by the script itself through a call to the TOOL API function: Throw().
Part of the reason for the deep level of “cross connection” between the major TOOL components is historical, in that the original design and implementation of TOOL was based on the BOB engine. Even though the major objects were designed based on broad functional areas, there were some areas where the separation could not be performed completely (just the same as in one cannot completely separate memory from the CPU in a “real” computer system). Even so, I’m hopeful that you will find my repackaging of the BOB project in to this particular set of classes is sensible and cohesive.
Looking more closely at what is contained in the VMCore, you can see that “at the bottom” of the TOOL system is a VMCoreVirtualChip (declared in VMCoreVirtualChip.h). This class is the “CPU” for the TOOL system. In this class, you will find such things as a vector for the interpreted code, the program counter, the stack and frame pointers for the machine, as well as the main heap storage areas.
At the next level out, there is the VMCoreVirtualMachine class (declared in VMCoreVirtualMachine.h) which fills the role of managing the TOOL heap and acts as the main memory manager class for the TOOL system. That being the case, this class will also work with the VMCoreVirtualChip to define and manage a simple dictionary of all variables and literals contained within a script program. Derived from the VMCoreVirtualMachine class is the VMCoreVirtualOpSys class. This class is where all the intrinsic functions (essentially the entire API to the TOOL engine) are defined and implemented. The VMCoreVirtualOpSys is just that, it is the “operating system” that is offered by the TOOL engine and is also the class where extensions to the engine are located. When you would like to add a new intrinsic function to your version of TOOL, you will likely implement it in the VMCoreVirtualOpSys class.
Still continuing on our journey out from the TOOL core, you’ll see that there is a VMCoreCompiler class which is inherited from the VMCoreParserScanner class. The parser-scanner object is the one that will scan and tokenize the script text, while the compiler will drive the scanner as well as operate against the scanner-parser output to assemble the array of op-codes that will be executed when the script is run.
At the highest level of TOOL, is the VMCoreInterpreter class that is responsible for executing the vector of compiled code that was produced by the VMCoreCompiler. The interpreter acts as a large switching-scanner. It scans the op-codes produced by the compiler, and then based on the op-code detected, will perform a specific set of actions.
How does the TOOL Engine work?
TOOL is implemented as a hybrid of a compiler and an interpreter (an inter-piler?). When a function is defined, it is compiled into instructions for a stack-oriented, byte-code machine. When the function is invoked, those byte-code instructions are interpreted. The advantage of this approach over a straight interpreter is that syntax analysis is done only once, at compile time. This speeds up function execution considerably and opens up the possibility of building a run-time-only system that doesn't include the compiler at all. In fact, in some of my scripts (which are not overly complex, but are not all that simple either), I’ve gotten over 150 complete executions a second through the system. Your mileage may vary of course, but TOOL, while nowhere near as fast as compiled code is no slouch either.
The virtual machine that executes the byte-codes generated by the TOOL compiler has a set of registers, a stack, and a heap. These constructs are contained in the VMVirtualChip class. All instructions get their arguments from and return their results to the stack. Literals are stored in the code object itself and are referred to by offset. Branch instructions test the value on the top of the stack (without popping the stack) and branch accordingly. Function arguments are passed on the stack, and function values are returned on the top of the stack. We’ll examine this concept again in more depth when the techniques for extending the TOOL engine are discussed.
In TOOL script classes, all member functions are virtual. This means that when a member function is invoked, the interpreter must determine which implementation of the member function to invoke. This is done by the SEND op-code, which uses a selector from the stack (actually, just a string containing the name of the member function) with the method dictionary associated with the object's class to determine which member function to use. If the lookup fails, the dictionary from the base class is examined. This continues, following the base-class chain until either a member function is found or there is no base class. If a member function is found to correspond to the selector, it replaces the selector on the stack and control is transferred to the member function, just as it would have been for a regular function. If no member function is found, an error is reported and the interpreter aborts. One extension I’ve added to this mechanism is a “cast call” with a syntax of “*->”. What this call does is to allow the script-writer to “cast” the class to a specific one of its base classes (named on the left hand side of the operator) and then to invoke the selector named on the right hand side of the operator. Several examples of this operator in action can be found in the script file “coreclasses.tool”. This operator serves two primary purposes. One is that it makes the call to a base class explicit, which is especially useful if multiple parent classes share the same function name, and also serves to comment to the script code as to what the author’s specific intentions are with regards to the execution of that class method.
TOOL variable types have been extended to support the following basic data types: Longs, Doubles, Bytes, Strings, Tokenizers, ByteArrays, Vectors, Classes, DateTimes, Queues, Maps, Stacks, WaitObjects, NT Kernel-handles, DWORDs, NT-File handles, File-Find handles, zib-files, SQLite databases, ODBC connections, and nil. Internally, the interpreter uses four more types: classes, compiled bytecode functions, built-in function headers, and variables. Wherever a value can be stored, a tag indicates the type of value presently stored there. This is the “v_type” field on the VMVariant class, declared in “VMCoreGlobal.h”.
Objects, vectors, and bytecode objects are all represented by an array of value structures. In the case of bytecode objects, the first element in the vector is a pointer to the string of bytecodes for the function, and the rest are the literals referred to by the bytecode instructions. Class objects are vectors, where the first element is a pointer to the class object and the remaining elements are the values of the nonstatic member variables for the object. Built-in functions are just pointers to the C functions that implement the built-in function. Variables are pointers to dictionary entries for the variable. There is a dictionary for global symbols and one for classes. Each class also has a dictionary for data members and member functions.
In addition to the stack, TOOL uses a heap to store objects, vectors, and strings. The current implementation of TOOL uses the C heap and the C functions malloc and free to manage heap space and uses a compacting memory manager. However, I must admit that the heap management in this version of TOOL is not completely garbage collected. This is due to the evolving nature of the engine and the fact that some of the new data types are actually C++ classes managed by the VMVariant object. This is why the TOOL API call “FreeObject” has crept into the TOOL API. This function is included so that the heap allocated C++ class can be cleaned up when the script is done using it. I recognize fully that this is a weakness in the current implementation, and one that I do plan to address in a newer version of the engine. But, for now, I’ve gotten used to the “evil of it” and have not therefore made a priority out of addressing the shortcoming of the current implementation.
Operator Overloading
In the TOOL interpreter, an operator can have several different meanings based on the LHS/RHS values for the operator. Consider the case of the array index operator “[]”. In this case, the operator can have several different interpretations. It should perform one action if the LHS is a Map object, and a complete different operation if the LHS is a Vector object. A similar approach is needed for mathematical operations, such as “+” or “-” where if the operation is performed on variables of different types, then one of them must be “promoted” to the other variable type. The interested reader is encouraged to study the VMInterpreter’s handling of the OP_ADD and OP_VREF op-codes for more details on how to extend operator overloading. This is an important topic to have a grasp of if you wish to add more variable types to the TOOL engine and wish to have specific behaviors for different “standard operators”.
Function Overloading
In the TOOL API, intrinsic functions can be “overloaded” in that they can be coded to accept different parameter counts and types. One example of this (and there are many others) is the HandlerNewDateTime() method in the VMCoreVirtualOpSys class. This handler is coded to accept either 1 or 6 arguments. If one argument is passed, then its type is examined to verify that it is a DT_DATETIME type. If six arguments are passed, then each of these arguments are examined to verify that they are DT_INTEGER types.
The most extreme implementation of this feature is found in the function HandlerStringFormat() which is an implementation of an elliptical function; meaning that it can accept any number and type of arguments, which is what you’d expect from a sprintf type of function.
This flexibility in defining functions is a natural outcome of the way the interpreter pushes function arguments on to the stack. The interpreter has no information regarding the number and type of arguments needed by any intrinsic function. It will simply push all arguments found for the call on to the stack. This feature enables any intrinsic function to be coded to accept all types of arguments from the interpreter.
There is a “flipside” to this flexibility however, and that is that all intrinsic functions should perform sanity checks on all arguments and argument counts to verify that what it is about to operate on is what is expected in the function.
All Ashore, That’s Going Ashore
Well that’s about a whirlwind of a tour that I can provide without getting completely bogged down in all the details that bring TOOL together. Hopefully, I have provided enough of an introduction to the lay of the land in the TOOL engine so that you can have some solid ground to stand on while you analyze the inner workings of TOOL. However, take comfort in knowing that you can easily use and extend TOOL without knowing how all the inside pieces and parts work. I just thought I’d show your around the project a bit, so that when you go to kick the tires on this beasty yourself, you’ll know where to find the fenders.
Programming in TOOL
The examples below present a simple example program; a function for computing factorials written using the TOOL language. Note: Readers are also encouraged to review the sample scripts included with this distribution.
Factorial( iValue )
{
return( ( iValue == 1 ) ? 1 : ( iValue * Factorial( iValue - 1 ) ) );
}
Run( ;iLoop )
{
iLoop = Long( 0 );
for ( iLoop = 1; iLoop < 10 iLoop++ )
{
Echo( StringFormat( "Factorial for % is %", iLoop, Factorial( iLoop ) ) );
}
}
You can see that this program looks a lot like its C counterpart. The only noticeable difference is the lack of a declaration for the type of the parameter iValue in the arguments passed to the Factorial function and for the return type of this same function. This is due to the fact that variable types do not need to be declared in TOOL; although it is still considered "good style" to declare and type all variables for clarity and maintenance purposes. A side effect of this is that any variable can take on a value of any type.
Other points worth noticing in this first sample are the variables declared in the Run function's formal parameter list. Here, you can see that the variable iLoop follows a semi-colon. This syntax is used to define the iLoop variable as being in "local scope" to the Run function. By default all variables in TOOL scripts adopt global scope unless they are declared using this specialized local scope syntax.
Important note: Even though TOOL is a weakly typed language from the script writer's perspective, the TOOL interpreter will type-check the type of a variable passed into its routines. If the variable is not of the right type, the interpreter will halt the execution of the script.
Again, what you should notice primarily is how much the example program above looks a lot like a similar program written in C. Also, notice that the program uses the StringFormat function to create a formatted string, and the Echo function to display the results. These functions will be explained in greater detail later in this document. The StringFormat function in TOOL prints each of its arguments in succession into an output string. It is capable of printing arguments of any type and automatically formats them appropriately.
In addition to supporting C-like expressions and control constructs, TOOL also supports C++-like classes. Again, since TOOL is a typeless language, the syntax for class definitions is somewhat different from C++, but it is similar enough that it should be easy to move from one to the other.
The next example shows a simple class definition that defines a class called Foo with members m_A and m_B and a static member called m_Last as well as a static member function GetLast. Unlike C++, it is not necessary to declare all member functions within the class definition; only the static member functions need be declared. It is necessary, however, to declare all data members in the class definition.
class Foo
{
m_A;
m_B;
static m_Last;
static GetLast();
}
Foo::Foo( AValue, BValue )
{
m_A = AValue;
m_B = BValue;
m_Last = this;
return( this );
}
Foo::GetLast()
{
return( m_Last );
}
As in C++, new objects of a class are initialized using a constructor function, which has the same name as the class itself. In this example, the constructor takes two arguments, which are the initial values for the member variables m_A and m_B. It also remembers the last object instance of the type Foo created in the static member variable m_Last. Lastly, the constructor returns the new object. For those of you not familiar with C++, the variable this refers to the object for which the member function is being called. It is an implicit parameter passed to every non-static member function. In this case, it is the new object just created.
In TOOL, all class data members are implicitly protected. The only way to access or modify the value of a member variable is through a member function. If you need to access a member variable outside a member function, you must provide access to member functions to do this.
We'll continue to refine the Foo class below to show how to set the value of a member variable. Finally, we'll show a member function that displays the numbers between m_A and m_B for any object of the Foo class type, and a Run function that creates some objects and manipulates them. The new operator creates a new object of the class whose name follows it. The expressions in parentheses after the class name are the arguments to be passed to the constructor function.
Foo::GetAValue()
{
return( m_A );
}
Foo::GetBValue()
{
return( m_B );
}
Foo::SetAvalue( NewAValue )
{
m_A = NewAValue;
}
Foo::SetBValue( NewBValue )
{
m_B = NewBValue;
}
Foo::GetSpan()
{
return( m_B - m_A );
}
Run( ;poFoo1, poFoo2 )
{
poFoo1 = new Foo( 1 , 2 );
poFoo2 = new Foo( 11, 22 );
Echo( "Foo1 Span Is: " + poFoo1->GetSpan() );
Echo( "Foo2 Span Is: " + poFoo2->GetSpan() );
}
TOOL also supports an inheritance model similar to the Java language, in that it allows one class to be derived from another. The derived class will inherit the behavior of the base class and possibly add some behavior of its own. TOOL only supports single inheritance; therefore, each class can have at most one base class. The next code example defines a class Bar derived from the base class Foo defined earlier.
The class Bar will have member variables m_A and m_B inherited from its parent Foo as well as the additional member variable m_C. The constructor for Bar needs to initialize this new member variable and do the initialization normally done for objects of class Foo. The example below illustrates how this is done:
class Bar : Foo
{
m_C;
}
Bar::Bar( AValue, BValue, CValue )
{
Foo*->Foo( AValue, BValue );
m_C = CValue;
return( this );
}
This example illustrates another difference between TOOL and C++ objects. In C++, constructor functions cannot be called to initialize already existing objects. This is allowed in TOOL, however, so the Foo constructor can be used to do the common initialization of the Foo and Bar classes. Careful readers may notice the *-> operation used in the Bar constructor. In TOOL, this is called the "cast-call operator". What this operator does is to cast "this" to the base-class-type named on the left-hand-side of the operator, and then invoke the function on the right-hand-side of the operator in the named base class. This syntax is especially useful for function overloading in long class hierarchies because this can be explicitly cast to the proper parent class type before the invocation of the function call.
A Quick Mention of Programming Style
Since TOOL is a weakly typed language, meaning that the type of variables is not required to be specified, I found it useful to use some means of classifying the data-type of a script variable (for the purpose of making the code more understandable to others). For this reason, a simplified form of Hungarian Notation is used in all script samples included with this article. Following are samples of the notation used:
Data Type Hungarian "Marker" Example
String s sMyString
Number i iValue
Object o oValue
I don’t mean to promote Hungarian Notation in any specific way, as I know that just as many folks don’t like it as I do, I simply find it to be a useful when striving to understand large projects.
More on a Suggested Programming Style
TOOL is a weakly typed language meaning that the type of variables does not have to be specified. Therefore, some means of classifying the data stored in a variable (for the purpose of making the code more understandable to others) would be of great benefit. Therefore, a simplified form of Hungarian Notation is encouraged in all TOOL program scripts. Following are samples of the notation used:
|
Data Type |
Suggested Marker |
Example |
|
String |
s |
sText
|
|
Long |
l or i |
lNumber
|
|
Double |
dbl |
dblValue
|
|
Byte |
b |
bFlag
|
|
Stack |
ostk |
ostkOfItems
|
|
Vector |
ovec |
ovecOfItems
|
|
Map |
omap |
omapOfItems
|
|
DateTime |
dt |
dtNow
|
|
Queue |
oqueue |
oqueueOfItems
|
|
Handle |
h |
hFile
|
|
Color |
clr |
clrBlack
|
|
DWORD |
dw |
dwValue
|
|
Database Connection |
oDB |
oDBSource
|
|
Calculator |
ocalc |
ocalcFormula
|
|
Class Instance |
po |
poFoo
|
Creating and Allocating TOOL Variables
There are several types of variables in TOOL. They are all listed below along with the TOOL call for creating the variable. Readers familiar with C++/C#/Java can think of these API calls as calls to a constructor for a variable.
Vector();
String( 255 );
String( "Hello" );
Queue();
Stack();
Map();
Tokenizer( sToTokenize, sDelimiter );
ByteArray();
ByteArray( Long( 25 ) );
ByteArray( Long( 25 ), Long( 5 ) );
Long();
Long( 50 );
Handle( 0 );
DateTime();
DateTime( dtOther );
DateTime( 2004, 01, 01, 12, 15, 0 );
Color();
Color( 0, 0, 0, );
Color( clrOther );
DWORD();
DWORD( dwOther );
DWORD( Long( 5 ) );
Database();
Calculator();
Double();
Double( Long( 6 ) );
Double( dblOther );
Double( "123.45" );
Byte();
Byte( bOther );
Byte( Long( 10 ) );
Page();
Table();
Color();
Database();
Calculator();
NullValue();
MiniDatabase();
UnZipper();
ZipMaker();
FileBuilder();
DelimitedFile();
ExcelExporter();
TOOL Run Time Type Checking
Since TOOL is a relatively weakly-typed language, as was explained previously in another topic; while at the same time the TOOL interpreter validates that all arguments passed to the TOOL API functions are of the proper type, TOOL offers an entire suite of RTTI (Run Time Type Identification) functions for script writers in order to assist them in writing solid and reliable script programs.
Use of these functions allows the creation of more robust scripts due to the argument validations that can be performed on all variables. An additional use of these functions may be to write scripts that will conditionally branch based on the type of variable being tested. There will be examples of both types of usage later in this document. The complete list of RTTI functions is listed below:
IsNull( oVarToTest );
IsClass( oVarToTest );
IsVector( oVarToTest );
IsNumber( oVarToTest );
IsString( oVarToTest );
IsFile( oVarToTest );
IsKernelObject( oVarToTest );
IsHandle( oVarToTest );
IsDateTime( oVarToTest );
IsDWord( oVarToTest );
IsNTHandle( oVarToTest );
IsFindHandle( oVarToTest );
IsQueue( oVarToTest );
IsStack( oVarToTest );
IsHashTable( oVarToTest );
IsConstant( oVarToTest );
IsConstCast( oVarToTest );
IsDouble( oVarToTest );
IsByte( oVarToTest );
IsByteArray( oVarToTest );
IsDatabase( oVarToTest );
IsReportPage( oVarToTest );
IsPageRegion( oVarToTest );
IsPageTable( oVarToTest );
IsColor( oVarToTest );
IsMiniDatabase( oVarToTest );
IsUnZipper( oVarToTest );
IsZipMaker( oVarToTest );
IsMiniRowSet( oVarToTest );
IsFileBuilder( oVarToTest );
IsDelimitedFile( oVarToTest );
IsExcelExporter( oVarToTest );
Note: See the class CRunTimeTypeInfo in the file ToolBox.Tool for a class that pulls all the RTTI APIs into a single TOOL class. The class sample shown is the base class for all TOOL framework classes, since all TOOL classes use variable type checks to insure the proper type of variables are passed to TOOL API functions from the TOOL framework.
TOOL Date Time Operations
Data processing and reporting often needs to deal with dates or date ranges. In fact, date time processing is likely to be "second most common" for data processing/reporting data types, with only string data being more frequently used. Because DateTimes are such a common requirement, TOOL offers a built in data type for that type of value; and also offers a set of API functions that offer the most common types of operations used for DateTime based values. These functions are listed below and a class wrapper for this family of functions can be found in the class called CDateTime which can be found in the ToolBox.Tool file.
dtNow = GetDate();
dtLater = DateAdd( sModPart, dtBase, iValue );
dtEarly = DateSub( sModPart, dtBase, iValue );
iPart = DatePart( sDatePart );
dtNew = DateSetHourPart( dtToChange, iHour );
dtNew = DateSetMinsPart( dtToChange, iMins );
dtNew = DateSetSecsPart( dtToChange, iSecs );
dtNew = DateSetHMS( dtToChange, iHour, iMins, iSecs );
sText = DateToString( dtDate );
dtNew = StringToDate( sText );
bLess = DateCompare( dtBase, dtOther );
iSpan = DateDiff( dtBase, dtOther );
TOOL File and Directory Operations
Directory Related Functions
Directory operations are very common for certain classes of data processing applications. To enable these types of jobs, TOOL includes several functions specific to those related tasks. The following classes illustrate the use of these types of functions.
The first TOOL class to study for this section of the TOOL API is found in the ToolBox.Tool file called CCurrentDirectory which implements a class wrapper over the TOOL functions related to manipulation of the current process directory. The current process directory is typically set by those scripts that are going to operate primarily from a single directory and use that directory as the "default path root" for all file-related operations.
The next directory related TOOL class wraps all those functions that actually manipulate the directory structure of a disk. There are TOOL API functions for moving, copying, comparing, renaming, and deleting entire directory trees on hard disks. See the CDirectory class in the ToolBox.Tool file for more information.
Directory and File Traversal Related Functions
Certain data transport tasks need to periodically scan file and directory structures as part of their processing. To support these types of jobs, TOOL offers a set of functions for traversal of directories and files. Class wrappers for these types of functions can be found in the ToolBox.Tool file in the CFind class.
For a practical example that uses these classes, refer to the simple network file backup program found in the TOOL Network Drive Operations topic. That example will demonstrate how TOOL can be put to use for purposes related to system administration in addition to tasks dedicated to data I/O, transformation and reporting.
The fact that TOOL can be used for various tasks outside of strictly data transport and reporting, adds additional value to the TOOL language in that the TOOL system has enough facilities included that allow it to be used for all types of operations.
File Related Functions
The TOOL API also offers file related functions for working with system files. TOOL provides a set of functions for inspecting file attributes, another set of functions for performing both text-based file I/O and for serialization of TOOL variables, ini-file I/O and file parsing. Classes that illustrate each of these portions of the API can be found in the ToolBox.Tool file.
See the CFileInfo in the ToolBox.Tool file for a class that implements a class wrapper for the file attributes and file management APIs built into the TOOL API. See the CTextFile class in the ToolBox.Tool file for a TOOL class that encapsulates all the TOOL API functions for text-file I/O.
Finally, see the CToolDataFile class in the ToolBox.Tool file which provides a wrapper over the TOOL API for the serialization of TOOL variable values. The class operates with a vector interface for reading and writing TOOL variables. The way TOOL serialization currently works, is that the script developer will "load a vector" with all the variables they want to save into a TOOL data file. The vector elements will be written in order to the data file. For restoring the values, the script simply reads back the vector data from the file.
Note, that at this time, collections of data (stacks, queues, maps, etc.) can not be serialized automatically. However, if demand warrants this type of functionality, it can easily be added to the TOOL run-time engine.
The final specialized area of TOOL file I/O processing is with regards to INI file processing. Since INI files are a convenient and easy to use method of storing simple hierarchies of configuration data, they are still a prevalent method of storing application parameters. So, as a convenience, TOOL also offers a set of functions for INI files. A class that provides a wrapper over these API functions is the CIniFile class found in the ToolBox.Tool file.
TOOL Network Drive Operations
TOOL offers a pair of functions that are related to connecting to network drives. There is one function to connect to a networked shared directory and another to disconnect from the same. The sample class named CNetworkDrive in the ToolBox.Tool file shows a class wrapper for these TOOL API functions.
In the CNetworkDrive class, you can see the straight-forward manner in which TOOL scripts can connect to network drives. There is another interesting item in this sample class however, and it is the Connect method; specifically, the test for:
sDriveLetter = "ERROR";
This shows another overloaded operator for string operations where TOOL string type variables can be directly tested for equality in addition to the StringCompare function. As an example of how to use the network drive class (and other classes) included in the ToolBox.Tool file, the following program shows how a TOOL program could be written to perform a network data file backup operation:
Run()
{
CNetworkDrive poNetOut = new CNetworkDrive();
if ( poNetOut->Connect( "\\\\MyServer\\BackupShare",
"Admin",
"Password" )
{
sNetDrive = poNetOut->GetDriveLetter();
for ( int iLoop = 0; iLoop < 2; iLoop++ )
{
if ( iLoop == 0 )
{
BackUpSingleServer( sNetDrive,
"Workstatation1",
"ImportantWork" );
}
else
if ( iLoop == 1 )
{
BackUpSingleServer( sNetDrive,
"Workstatation2",
"FinanceData" );
}
else
if ( iLoop == 2 )
{
BackUpSingleServer( sNetDrive,
"Workstatation3",
"CustomerData" );
}
}
}
}
BackUpSingleServer( sNetDrive,
sServerName,
sServerShare,
; sTargetCopyPath,
sLastBackUpPath )
{
CNetworkDrive poNetIn = new CNetworkDrive();
sTargetCopyPath = StringCopy( sNetDrive );
sLastBackUpPath = StringCopy( sNetDrive );
sTargetCopyPath += StringFormat( "\\%.Backup",
sServerName );
sLastBackUpPath += StringFormat( "\\%.Prev.Backup",
sServerName );
if ( poNetIn->Connect( StringFormat( "\\\\%\\%",
sServerName,
sServerShare ),
"Admin",
"Password" ) )
{
sInDrive = poNetIn->GetDriveLetter();
CDirectory poDir = new CDirectory( sInDrive );
poDir->CopyDirectory( sTargetCopyPath,
sLastBackUpPath );
if ( poDir->CompareToOther( sTargetCopyPath ) )
{
}
else
{
Echo( FormatString( "Failure during backup of \\\\%\\%.\r\n",
sServerName, sServerShare ) );
}
poNetIn->Disconnect();
}
else
{
Echo( FormatString( "Failed to backup: \\\\%\\%.\r\n",
sServerName, sServerShare ) );
}
}
The TOOL program above illustrates how to instantiate and make use of TOOL language class objects in a program. This demonstration program also shows how TOOL can be put to multiple uses in an organization in addition to its use as a data-management and reporting system by showing how a simple network backup program can be written in TOOL.
TOOL Registry Operations
Since TOOL developers may need to have their scripts interact with the system's registry, TOOL offers APIs for interacting with this aspect of the host system as well. The support offered by TOOL is enough to use the Registry for I/O of strings and numbers. Since nearly all TOOL data types can be converted to these two variable types, the TOOL API is so constrained for simplicity and ease of use (a recurring design goal for TOOL).
If additional functionality is required, then it will not be a difficult task to extend TOOL to support any new requirements with regards to operating with the system registry.
As is typical for this document, a class wrapper for this section of the TOOL API is found in the ToolBox.Tool file and in the TOOL class called CRegistry.
TOOL System Environment Operations
Periodically, in certain types of system instillations, the operating system's environment variables are used for certain purposes. For operations in these types of installations, TOOL provides functions that allow TOOL scripts to read/write to the system environment variables. The TOOL class called CEnvironment in the ToolBox.Tool file illustrates this portion of the TOOL API.
TOOL Process Control Functions
TOOL offers functions for starting, checking-on, and monitoring other processes. This allows TOOL to inter-operate with already existing programs, and/or to coordinate the activities of the system via the control over processes which are external to the TOOL run-time engine. TOOL can simply spawn a secondary process with the following call:
ProcStartNoWait( sProcessCommandLine );
When this API is invoked, the TOOL run-time will forward the process-command-line to the operating system for execution and then will return immediately, so as to not wait for the newly-started process. A related function is to query the operating system to determine if a particular process is running in the operating system where the TOOL runtime is hosted. This call is shown below:
IsProcessRunning( sProcessName );
These two functions can allow TOOL to be utilized as a simple-to-use watchdog program (a program that keeps another process running). The following example illustrates this concept:
WatchDog( sProcessName, sProcessCmdLine )
{
while ( 1 )
{
if ( !IsProcessRunning( sProcessName ) )
{
ProcStartNoWait( sProcessCmdLine );
}
Sleep( 5000 );
}
}
Another interesting TOOL API for starting external processes is:
StartLoggedProcess( sProcssesCmdLine, sFullPathToLogFile );
This API will start up a process and then will capture all data from that process' "stdout" to a log file passed to this function. This means of starting processes is very useful for running simple utility programs from the TOOL system, and then capturing the output from those utilities to a file for future trouble-shooting or analysis.
The next TOOL API for starting processes is one that allows a TOOL script to get a process handle from the operating system for the process just started. This handle then can be used as a "wait handle" so that TOOL can wait for the process to complete prior to continuing.
hHandle = StartProcessGetHandle( sProcessCmdLine );
....
ReleaseHandle( hHandle );
The ReleaseHandle call should be called from the script when interest in the process handle has ceased. The final process control function offered by TOOL is one that allows TOOL scripts to monitor and analyze the output of a slave process. When this API is invoked, TOOL will start and "attach to the output from" a process. What this allows is for all output produced by the slave process' "std output" and/or "std error" streams to be read and analyzed by the TOOL script. The API function that does this follows:
ProcStartErrorCheck( sRunLine, oVecErrorList );
The following function illustrates how to use this TOOL API function:
StartProgramAndCheckResults( sCommandLine; oVecErrorList, bSuccess )
{
oVecErrorList = Vector();
bSuccess = Byte( 0 );
VectorAddItem( oVecErrorList , "ERROR" );
VectorAddItem( oVecErrorList , "UNAVAILABLE" );
VectorAddItem( oVecErrorList , "DOES NOT EXIST" );
VectorAddItem( oVecErrorList , "INCORRECT SYNTAX" );
VectorAddItem( oVecErrorList , "NO SUCH FILE OR DIRECTORY" );
VectorAddItem( oVecErrorList , "COLUMN DOES NOT ALLOW NULLS" );
VectorAddItem( oVecErrorList , "INVALID OBJECT NAME" );
VectorAddItem( oVecErrorList , "INVALID COLUMN NAME" );
VectorAddItem( oVecErrorList , "CHOOSE ANOTHER" );
VectorAddItem( oVecErrorList , " LEVEL" );
VectorAddItem( oVecErrorList , "FAILED" );
VectorAddItem( oVecErrorList , "FILE NOT FOUND" );
bSuccess = ProcStartErrorCheck( sCommandLine, oVecErrorList );
if ( !bSuccess )
{
VectorClear( oVecErrorList );
FreeObject( oVecErrorList );
Throw( StringFormat( "The program % returned an error.\r\n",
sCommandLine ) );
}
VectorClear( oVecErrorList );
FreeObject( oVecErrorList );
}
TOOL Inter-Process Synchronization Primitives
TOOL offers a complete set of process-synchronization variable-types. These variable types are important in those installations where TOOL script programs will coordinate actions with other external programs/processes through the signaling of these cross-process variables. These variables will also be used internally in TOOL multi-threaded-scripts.
The explanation of the appropriate use of these variable-types is beyond the scope of this document. For that type of information, the reader is referred to any reference that deals with multi-threaded programming techniques.
A class wrapper for each of the TOOL synchronization variable types is found in the ToolBox.Tool file. The first variable type you may want to examine is a class wrapper for a semaphore, which as you may recall is an inter-process variable that has a "count" associated with it. The variable can be "acquired" simultaneously by as many processes/threads as the "count" supports. See the CSemaphore class in ToolBox.Tool for that class.
The next type of inter-process variable supported in TOOL is the Mutex. This type of variable can only be "acquired" by a single process and/or thread at any time. This type of variable is a "mutual exclusion" gate and serves as a global "single threaded control" across the entire operating system. The TOOL class wrapper for that variable type is called CMutex and can be found in the ToolBox.Tool file.
The final type of inter-process variable supported in TOOL is an Event. An event can be used to "send signals" between processes and/or threads for purposes of coordinating activities between them. Properly understood and used, Events can lead to powerful interactions between completely separate processes. The CEvent class in the ToolBox.Tool file implements an event class.
Now that we've covered the creation and simple management of the inter-process variables supported in TOOL, the next topic (naturally) is how to use them in "wait sets" the use of which allows TOOL to coordinate its actions with other external processes. The CWaitSet class in ToolBox.Tool shows one possible way to use the TOOL wait-related.
Working with TOOL Collection Objects
TOOL offers several data types for managing collections of data. Included in the TOOL installation are sample programs that illustrate how to work with each collection type. It is hoped that the reader will start to become more familiar with class declarations in the TOOL language at the same time that they are being introduced to the collections API functions themselves.
Determining Element Counts of any Collection
TOOL offers a SizeOf function for determining the number of elements in any collection type variable. The following types of variables can be passed to this function:
- String
- Vector
- Stack
- Queue
- Map
- ByteArray
as follows:
lElements = SizeOf( oCollectionTypeVar );
TOOL String Operations
Strings are probably the most common data types used in nearly all data processing tasks. Due to that situation, string manipulation is probably the most common types of operations performed in data processing.
Because of this, the TOOL API for Strings is the "largest single section" dedicated to a single data type. The complete list of String-related API functions is listed below. Since the String-related API is as rich as it is, the sample class is likely to be the largest sample included in the samples provided with this distribution.
iResult = StringCompare( sOne, sTwo );
sResult = FillString( sBase, iStartAt, iSpanFor, sFillWith );
iIndex = IndexOf( sBase, sToFind );
sResult = TrimString( sToTrim, sType );
iLength = StringLength( sText );
sResult = ToLower( sInput );
sResult = ToUpper( sInput );
sResult = SubString( sInput, iStartAt, iLength );
sResult = StringGetAt( sInput, iIndex );
sResult = StringConcat( sInput, sToAppend );
sResult = StringReplace( sInput, sToReplace, sReplaceWith );
sResult = StringFormat( sFormatString, ... );
sResult = StringBoundedBy( sInput, sLowerToken, sUpperToken );
sResult = BoundedStringReplace( sIn, sLowToken, sUpToken, sNewText );
sResult = TrimQuotes( sInput );
sOutput = StringCopy( sInput );
sResult = PutQuote( sInput );
sResult = PadLeft( sInput, iPadSize );
sResult = PadRight( sInput, iPadSize );
bNumber = IsStringNumeric( sInput );
A rather large TOOL class that implements a class wrapper over the TOOL String-based API functions is the CString class found in the ToolBox.Tool file. That class contains a large amount of functionality (since there are so many types of operations typically performed on the string type data), so it is the largest one presented so far in this document.
The class also illustrates a few of the string-to-other-data-type conversion functions offered by TOOL. Even so, it is a straight-forward class and as such should offer no difficulties to the reader.
Another very important string function offered by TOOL is the "StringFormat" function. This elliptic function is very useful in building a single string from any number of data values appended into a format string. See the examples below:
sResult = StringFormat( "Today's data is: %", GetDate() );
sResult = StringFormat( "The value mapped in the under the key % is %",
iKey,
MapFindByKey( oMap, iKey ) );
With the StringFormat function, the format string contains a '%' character at each position where the 'string value' to the next format argument is to be inserted in the output string.
To 'escape' the '%' character and output an actual percent sign, use '%%' which will output a single percent character into the resultant string at that location.
While there is no practical limit to the number of arguments that can be passed to the StringFormat function, the resultant string is limited to 4-KBytes in length. Another very important String operator is the '+' sign which can also be used for string concatenation, as follows:
sString = "The quick brown fox";
sString += "jumped over the lazy dog";
sString += "and the cow \"jumped\" over the moon";
In the example above, you can also see that the '\' character can be used to "escape" a double-quote that should be embedded in the resultant string.
Tokenizing Strings in TOOL
To perform simple tokenization on string type variables, TOOL offers a built in tokenizer. A class wrapper for TOOL's String tokenizer functions is found in the CTokenizer class in the ToolBox.Tool file.
TOOL Vector Operations
TOOL offers a set of functions specific to operating on Vector type variables. For all these functions, the first argument to each of these functions is the Vector object upon which the function should operate against. Each of these operations will be introduced below.
TOOL Vector objects can contain a "mixed collection" of objects, meaning that each of the elements stored in the collection can be of different types.
VectorSetAtIndex( oVector, iIndex, vValue );
VectorGetAtIndex( oVector, iIndex );
VectorAddItem( oVector, vValue );
VectorDelAtIndex( oVector, iIndex );
VectorClear( oVector );
See the CVector class in the file ToolBox.Tool for a TOOL class that provides a class wrapper for all the Vector related functions offered by TOOL. That example illustrates once again how to declare a TOOL class as well as offering a rather complete implementation of a Vector object wrapper for TOOL vector-related functions.
In examining the CVector class example, one can see that the Vector class not only provides a complete class wrapper to all TOOL vector related functions; but that it extends the CRunTimeTypeInfo class introduced in the discussion of RTTI related functions; and that calls to the base class are used to validate arguments passed prior to making calls into the TOOL engine. For each of these validation calls, you can see the use of the cast-call-operator.
Note: It is considered good form to validate all arguments prior to invoking the API functions if there is any doubt as to the type of variable being passed to the TOOL runtime engine. Adopting this philosophy will help to make your TOOL scripts more robust and reliable and will prevent abrupt halts of your scripts by the run-time should you pass incorrect variable types into the TOOL API functions.
Through class design techniques, it is possible to implement TOOL collections that are "type checked" as is shown in the extension to the CVector class called CNumberSet in the ToolBox.Tool file. This class shows an example of a type constrained collection that will only store a collection of numbers.
In the CNumberSet example class, one can see that the SetAtIndex and AppendItem class functions have been overridden so that RTTI checks can be performed on the arguments passed to the class functions. You can also see the cast-call-operator usage again in several of these function calls including the class constructor.
In addition, the CNumberSet class also illustrates the use of three more specialized Vector operations; specifically the Max, Min, and Avg TOOL API functions which operate against all the numbers contained in a Vector variable and return the appropriate value from the operation.
TOOL Byte Array Operations
One of the TOOL data collection variable types is the Byte Array variable type. This type is similar to the Vector type explained in another section of this document, with one very important difference: the Byte Array only stores arrays of bytes. This class is useful as a "buffer class" for forming particular arrangements of byte values.
In looking over the list of functions shown below, you can see some of the Byte Array related functions offered by the TOOL API. You can see that the ByteArrayAppend and ByteArrayInsertAtIndex functions have several different overloads and can work with several types of formal arguments. This type of function overloading is possible due to the stack-oriented design of the TOOL engine, and the technique is used when it makes sense to offer an overloaded implementation.
ByteArrayGetAtIndex( oByteArray, iIndex );
ByteArraySetAtIndex( oByteArray, iIndex, bValue );
ByteArrayAppend( oByteArray, bAByte );
ByteArrayAppend( oByteArray, sAString );
ByteArrayAppend( oByteArray, oAnotherByteArray );
ByteArrayInsertAtIndex( oByteArray, iIndex, bAByte );
ByteArrayInsertAtIndex( oByteArray, iIndex, sAString );
ByteArrayInsertAtIndex( oByteArray, iIndex, oAnotherByteArray );
ByteArrayDelAtIndex( oByteArray, iStartDeletAtIndex, iCountToRemove );
ByteArrayClear( oByteArray );
The class CByteArray in ToolBox.Tool file is a sample class wrapper for the Byte Array variable type API functions offered by TOOL. In the class example, you will again see the use of the cast-call-operators and RTTI functions being used. After a review of the Byte Array class wrapper, and through studying the other classes in the ToolBox.Tool file, you should start to get a pretty good idea as to how to construct a class in the TOOL language.
TOOL Stack Operations
TOOL offers a Stack object for those scripts that can benefit from such a LIFO collection type. Like the Vector type, the Stack can store various variable types in a single collection. A Stack is rather a simple collection type, so the number of API functions dedicated to Stack is more limited than for other TOOL collection types. See the list below:
StackPush( oStack, oToPush );
StackPop( oStack );
StackPeek( oStack );
StackClear( oStack );
Refer again to the ToolBox.Tool file for the CStack class, which is a simple TOOL Stack class wrapper as shown to illustrate the use of all the stack related functions in TOOL. As you can see, this Stack class is relatively simple due to the fact that the Stack related API functions are limited by the straight-forward nature of the Stack variable type.
TOOL Map Operations
Currently, TOOL offers only one type of associative container; the Map which allows a simple storage container for keyed-value pairs. The TOOL Map data type constricts the types of variables that can be used as keys into the collection; but like all other TOOL containers, the values stored in the container can be any TOOL value type, and multiple variable types can be stored in a single map. If an illegal key type is attempted to be used with the TOOL Map, the TOOL Run-Time-Engine will halt execution of the TOOL script.
Like other TOOL data collections, the Map container is designed to be simple and straight-forward, so the number of Map related functions is limited to those functions required to work with the data type. The list of Map related API calls is listed below:
MapRemoveKey( oMap, aKey );
MapFindByKey( oMap, aKey );
MapHasKey( oMap, aKey );
MapClear( oMap );
For an example implementation of a TOOL Map class wrapper, see the CMap class in the ToolBox.Tool file. In order to implement a CMap class that is constrained to a single key or value type, one would need to revise the IsLegalKeyType and/or the IsLegalValueType class methods to provide a more restrictive set of tests on the keys and values allowed to be stored in the underlying Map variable.
TOOL Queue Operations
Another classic data collection offered by TOOL is the Queue data type. The TOOL Queue is an implementation of the FIFO data collection commonly used in many programming projects. The entire Queue related API is shown below:
EnQueue( oQueue, oToPush );
DeQueue( oQueue );
QueueClear( oQueue );
The TOOL class implementation of a Queue class is the CQueue class in the Toolbox.Tool file. Like the Stack data type, the TOOL Queue implementation is straightforward and does not require a large amount of functions to implement. From the samples provided, you can also see how easy it is to work with a TOOL queue.
TOOL Application Runtime Environment
Parameters can be passed into the TOOL runtime context for use as "environment variables" for the duration of that TOOL program. TOOL scripts can gain access to these variables with the calls explained below.
The RunTool.exe program can be invoked with command-line-arguments that will be forwarded to a TOOL script's runtime context. See the following example for how this could be done:
For other TOOL programs that are running in a TOOL run-time environment, there are two additional API calls that can be used to get and set variables within the program context. For fetching a variable from the application server environment, use the API "GetAppEnvField". An important distinction between "GetAppEnvField" and "GetProperty" is that "GetProperty" will always return a string type variable, whereas "GetAppEnvField" can return variables of the following types:
- String
- Long
- Byte (boolean)
- Double
- DWORD
- DateTime
- ByteArray
depending on the type of the data that was stored in the application context with the call to the API "SetAppEnvField". (Note that the above list of variable types are the only ones that will be allowed to be stored in the application context through the "SetAppEnvField" call. If any other type of variable is passed to the "SetAppEnvField" function, a run-time error will result and execution of the TOOL script will be terminated.) The sample below shows how these API functions may be used.
SetAppEnvField( "THE_STRING", "Hello" );
SetAppEnvField( "THE_NUMBER", 1234567 );
SetAppEnvField( "FALSE_BOOL", Byte( 0 ) );
SetAppEnvField( "TRUE_BOOL", Byte( 1 ) );
SetAppEnvField( "DOUBLE_VAL", 1234.5678 );
SetAppEnvField( "NEW_YEARS", DateTime() );
xFetched = GetAppEnvField( "THE_STRING" );
xFetched = GetAppEnvField( "DOUBLE_VAL" );
See the TOOL class "CAppEnvironment" in the file ToolBox.Tool for a class that implements the application environment interface.
TOOL Database Operations
Since one of the primary purposes envisioned for TOOL is use in tasks related to data gathering, manipulation and storage of database data, TOOL must offer a database I/O system. The goal of this section of the TOOL API was to provide enough functionality to "get the job done" while attempting to avoid a lot of unnecessary complexity.
The interface method used by TOOL to connect to databases is the ODBC drivers offered by all major DBMS vendors. The purpose for this is that this level of interface is the lowest-common-denominator for all database connections (in fact even more 'advanced interfaces' are typically built as additional code layers over ODBC). In addition, it is a ubiquitous technology at this point and therefore will allow TOOL to "connect to virtually anything" on an equal basis.
Internally, TOOL has a sophisticated ODBC wrapper technology that allows access to database results to be processed without the TOOL programmer having to deal with the 'grungy details' of dynamically binding to the results of "any conceivable query" that are a part of TOOL jobs.
The database functions offered by TOOL are designed for a simple and straight-forward application of database I/O. The "command interface" between TOOL scripts and the connected databases is through SQL-strings passed to the DBMS via the TOOL-managed database connections. Again, the primary design consideration is for ease of use and portability between all database types/vendors rather than to build TOOL too tightly to a single DBMS vendor. The sample code below shows the simplest possible usage of the database API offered by TOOL:
TestDB(;oDB,iTimeOut,iColumnCount,iLoop,oValue)
{
oDB = Database();
iTimeOut = Long( 200 );
iColumnCount = Long( 0 );
iLoop = Long( 0 );
oValue = Long( 0 );
if ( DBOpen( oDB, "DSN", "DB_USER", "DB_PASS", "USE_DB" ) )
{
DBSetLogonTimeOut( oDB, iTimeOut );
DBSetQueryTimeOut( oDB, iTimeOut );
if( DBExecQuery( oDB, "select * from TABLE" ) )
{
if ( !DBIsEmptySet( oDB ) )
{
iColumnCount = DBGetColumnCount( oDB );
DBMoveToStart( oDB );
while ( !DBIsEOF( oDB ) )
{
DBFetchRow( oDB );
for ( iLoop = 0; iLoop < iColumnCount; iLoop++ )
{
oValue = DBGetFieldByIndex( oDB, iLoop );
if ( IsByteArray( oValue ) )
{
FreeObject( oValue );
}
}
DBMoveNext( oDB );
}
}
}
}
DBClose( oDB );
FreeObject( oDB );
}
Rather than present a class wrapper over the TOOL database API, a sample data replication program will be included in the following sample as it illustrates a more complete and 'real-world' application of this part of the TOOL API.
Run()
{
SetupGlobals();
if ( ConnectToSource() && ConnectToTarget() )
{
if ( PrepareSourceForDataMove() && PrepareTargetForDataMove() )
{
MoveData();
PostSourceDataMove();
PostTargetDataMove();
}
}
CleanUp();
}
SetUpGlobals()
{
iTimeOut = Long( 200 );
oSourceDB = Database();
oTargetDB = Database();
sSourceDSN = String( "ODBC.DSN" );
oSourceUID = String( "USER" );
oSourcePWD = String( "PASS" );
oSourceDB = String( "DBNAME" );
sTargetDSN = String( "ODBC.DSN" );
oTargetUID = String( "USER" );
oTargetPWD = String( "PASS" );
oTargetDB = String( "DBNAME" );
sSourceQuery = String( "select * from T_SOME_TABLE" );
sTargetTable = String( "T_TARGET_TABLE" );
}
CleanUp()
{
DBClose( oSourceDB );
FreeObject( oSourceDB );
DBClose( oTargetDB );
FreeObject( oTargetDB );
}
ConnectToSource()
{
if ( DBOpen( oSourceDB,
sSourceDSN,
sSourceUID,
sSourcePWD,
sSourceDB ) )
{
DBSetLogonTimeOut( oSourceDB, iTimeOut );
DBSetQueryTimeOut( oSourceDB, iTimeOut );
return( Byte( 1 ) );
}
return( Byte( 0 ) );
}
ConnectToTarget()
{
if ( DBOpen( oTargetDB,
sTargetDSN,
sTargetUID,
sTargetPWD,
sTargetDB ) )
{
DBSetLogonTimeOut( oTargetDB, iTimeOut );
DBSetQueryTimeOut( oTargetDB, iTimeOut );
return( Byte( 1 ) );
}
return( Byte( 0 ) );
}
PrepareSourceForDataMove()
{
return( Byte( 1 ) );
}
PrepareTargetForDataMove()
{
return( Byte( 1 ) );
}
PostSourceDataMove()
{
}
PostTargetDataMove()
{
}
MoveData( ;oValue, iColumnCount, iLoop, sTargetInsert, bInTran )
{
iLoop = Long( 0 );
iColumnCount = Long( 0 );
sTargetInsert = String( 255 );
if ( DBExecQuery( oSourceDB, sSourceQuery ) )
{
if ( !DBIsEmptySet( oSourceDB ) )
{
iColumnCount = DBGetColumnCount( oSourceDB );
DBMoveToStart( oSourceDB );
while ( !DBIsEOF( oSourceDB ) )
{
if ( !DBIsOpen( oSourceDB ) )
{
Echo( "Source Connection Unexpectedly closed\r\n" );
return;
}
DBFetchRow( oSourceDB );
sTargetInsert = "insert into ";
sTargetInsert += sTargetTable;
sTargetInsert += " values ( ";
for ( iLoop = 0; iLoop < iColumnCount; iLoop++ )
{
oValue = DBGetFieldByIndex( oSourceDB, iLoop );
if ( IsNull( oValue ) )
{
sTargetInsert += "null ";
}
else
if ( IsNumber( oValue ) )
{
sTargetInsert += IntToString( oValue );
}
else
if ( IsString( oValue ) )
{
sTargetInsert += "'";
sTargetInsert += oValue;
sTargetInsert += "'";
}
else
if ( IsDateTime( oValue ) )
{
sTargetInsert += "'";
sTargetInsert += DateToString( "%x %X", oValue );
sTargetInsert += "'";
}
else
if ( IsDouble( oValue ) )
{
sTargetInsert += DoubleToString( oValue, 10 );
}
if ( iLoop < iColumnCount - 1 )
{
sTargetInsert += ", ";
}
}
sTarget += " )";
bInTran = Byte( 0 );
if ( !DBIsOpen( oTargetDB ) )
{
Echo( "Target Connection Unexpectedly closed\r\n" );
return;
}
if ( DBCanTransact( oTargetDB ) )
{
bInTran = Byte( 1 );
DBBeginTran( oTargetTB );
}
if ( DBExecQuery( oTargetDB, sTargetInsert ) )
{
if ( bInTran )
{
DBCommitTran( oTargetDB );
}
}
else
{
if ( bInTran )
{
Echo( StringFormat( "\r\nWARNING! Query\r\n % \r\n FAILED",
sTargetInsert ) );
DBRollbackTran( oTargetDB );
}
}
DBMoveNext( oSourceDB );
}
}
}
}
The above program illustrates a practical example of a data replication program written in TOOL. You can see that TOOL offers a database interface that is complete enough to complete better than 99% of the tasks required for many typical database I/O operations; even still managing to remain simple and straightforward. Again, the driving factor in the design of TOOL is to focus strictly on the tools required by a specialized data processing language while not getting bogged down in 'corner case special requirements', for most projects do not add significant value and only serve to complicate the task at hand in nearly all practical applications. There are some additional TOOL database API functions not covered in the example above. These are:
bIsAtBeginning = DBIsBOF( oDB );
DBMovePrev( oDB );
DBMoveToEnd( oDB );
DBMoveAheadBy( oDB, iMoveBy );
DBMoveBackBy( oDB, iMoveBy );
DBMoveFromStartBy( oDB, iIndex );
DBMoveFromEndBy( oDB, iIndex );
DBGetFieldByName( oDB, sFieldName );
TOOL Memory Management Functions
Freeing TOOL Objects
There are some TOOL variable types that must be disposed off when the script has completed use of them. These variable types are:
- Tokenizer
- Stack
- Queue
- Map
- Calculator
- ByteArray
- Vector
- Database
Note: Future versions of TOOL are expected to host an improved memory manager, that would eliminate the need to free TOOL variables, but for now, it is simple to destroy variables with a call to the API FreeObject as follows:
FreeObject( oSomeVar );
Note: If FreeObject is called with a variable type not requiring destruction, no problem will result.
That’s Great, But I Just Want to Scratch Around on the Surface
We’ve already discussed how to integrate TOOL into your application. Hopefully, you will agree that the task is rather easy to pull off. The next natural step is to discover how to extend TOOL to suit your specific purposes. We’ve touched on a couple of these areas above, but here we’ll delve into the topic to the next level. There are countless ways to add on to TOOL, but in my experience, there are some types of extensions that could use some hints as to how to pull off an extension completely. The areas I’ll cover here are:
- Adding New Op-Codes
- Adding New Variable Types
- Adding New Intrinsic Functions
For these overviews, I will summarize the areas of the code base that will need to be extended. I don’t think I have space to do much more than that, so hopefully the information provided will be enough to get you started.
Adding new op-code to the system is among the most complex actions one can do to extend TOOL. Do not attempt to do this until you have a very firm grasp on all the implications of your action. I’ve done this once and twice and it involves a serious amount of head scratching. But if you must to this, then use the following guidelines to add new op-codes, the following changes are needed: declare the op-code in VMCoreGlobal.h at/around line 597; add an entry to the otab table in VMCoreInterpreter.cpp at/around line 100 (this table is used when dumping op-codes during traced/decoded runs); add a new switch/case for the new op-code in the interpreter’s main loop at/around line 349 and put any specific coded needed to handle the op-code in that case block; add new code to eject the op-code as needed in VMCoreScannerParser.cpp at/around line 180; if your new op-code will also add a new token to the system, then you’ll need to define that token in VMCoreGlobal.h at/around line 109; you’ll need to also extend VMCoreScannerParser.cpp at/around line 180; and finally add any appropriate code to the VMCoreCompiler to build op-code array values appropriate to the new token.
Adding new variable types to the TOOL system is another involved operation but not nearly as much as adding new op-codes. The key points to remember here are that you may need to extend the operation of certain interpreter op-code handlers so that they are aware of your new variable types as well as how to implement them. This alluded back to the section on operator overloading above. With that notice in mind, these are the steps required to add a new variable type to TOOL: Add a new data-type declaration in VMCoreGlobal.h at/around line 202; if your new data-type will be implemented as a C++ class, then forward declare that class in VMCoreGlobal.h at/around line 243; modify, as appropriate, the VMVariant (the variant class used for all TOOL variables) constructor in VMCoreGlobal.h at/around line 305; modify, as appropriate, the ResetValue() method in VMCoreGlobal.h at/around line 319; modify, as appropriate, the VMVariant union in VMCoreGlobal.h at/around line 369; modify, as appropriate, the VMVariant copy constructor in VMCoreVirtualOpSys.cpp at/around line 76; add a new intrinsic function to instantiate/construct the new variable type, see VMCoreVirtualOpSys.cpp at/around line 207 for how to declare the instantiation handler; also see the HandlerNewString() method in VMCoreVirtualOpSys.cpp at/around line 1841 for some information on how to implement the instantiation handler; add a new RTTI type handler for your new data type, see VMCoreVirtualOpSys.cpp at/around line 308 for how to declare your new type check function, see the method HandlerIsByteArray() in VMCoreVirtualOpSys.cpp at/around line 1234 for how to implement your new type checking function; and declare and add any new intrinsic functions specific to your new variable type, see VMCoreVirtualOpSys.cpp at/around line 238 for some examples of this step. If your new variable is subject to operator overloading, then you’ll also possibly need to extend the VMCoreInterpreter class op-code-switch handlers for logical, array index, and mathematical handlers as appropriate for your new variable type. The handlers that are likely to be affected (if any) are the following: OP_NOT; OP_NEG; OP_ADD; OP_SUB; OP_MUL; OP_DIV; OP_REM; OP_BAND; OP_BOR; OP_XOR; OP_BNOT; OP_SHL; OP_SHR; OP_LT, OP_LE; OP_EQ; OP_NE; OP_NE; OP_GE; OP_GT; OP_INC; OP_DEC; OP_VREF; OP_VSET.
Next, we’ll see how to add a new intrinsic function to TOOL. You’ll see that this is the easiest way to extend the engine out of all the ways discussed here. What you are doing when you do this is that you are adding a new function pointer to the interpreter's map of function pointers to intrinsic functions. Whenever the interpreter recognizes the name of an internal function, it builds up the call stack for it (putting all arguments on the “call stack” ordered from left to right) and then invokes the function via the pointer to it. First, you’ll need to declare the function, see VMCoreVirtualOpSys.h anywhere after line 100 for an example of this. You’ll quickly see that all intrinsic functions share a common signature. Next, you’ll need to add the handler to the internal list of intrinsic functions with a call to ConfigureBuiltInFunction(). See VMCoreVirtualOpSys.cpp starting at line 200 for many examples of this. The two arguments to the configure function are:
- the “name” of the function as it will appear in script files;
- the function pointer to your implementation of that handler.
Recall from the discussion above that the interpreter “built up a call stack” immediately prior to invoking the intrinsic function. The handler must then (if it were implemented with safety in mind) validate the number and types of values passed to the handler. Let’s examine HandlerStringCompare() in VMCoreVirtualOpSys.cpp at/around line 4311 for some examples of how to validate the inbound arguments. The parameter iArgc is the count of arguments the interpreter placed on to the call-stack for the function. So, if the intrinsic function has expectations about the number of arguments that should be passed, it should validate that fact first. You can see this is easily done with a call to VerifyRunLineArguments(). Next, if the function has expectations regarding the types of variables passed, it should validate those next. This can also be easily checked with a call to VerifyElementType(). The first parameter to this function is the “stack offset” of the parameter to check, and the second argument is the type of variable expected at that stack location. Recall that the interpreter will load the stack from “left to right” which means that the “first parameter to the function” will have the highest stack offset on the call stack and that the last argument passed to the intrinsic function will have a stack offset of zero. You’ll need to keep this fact in mind as you write your own verification code. Finally, if you need to return any value back to the interpreter from your function, you’ll need to do so calling the SetValue() function with your results. The interpreter expects that any return value from your function will be left on the stack after the function returns. This means that you will always return results to the stack location: m_poVirtualChip->m_psStackPointer[iArgc]. The interpreter also expects that the function will “erase the call frame” by adjusting the stack pointer appropriately. This means that the last line in any intrinsic function will likely be: m_poVirtualChip->m_psStackPointer += iArgc; which has the effect of “erasing the call stack”.
Writing TOOL Programs
I’ll refer the reader to the CoreClasses.Tool file for a tutorial on how to write programs in the TOOL language. This file contains a number of TOOL style classes that create class wrappers over the TOOL API. This set of classes, as well as the functions in this file, are my “test jig” for TOOL and as such exercise nearly all of the TOOL API in a single script. This script file then, serves as an excellent example of how to program in the TOOL language. In addition, it also offers a set of classes written in TOOL and which provides a set of tested and ready-to-use TOOL classes that can be put into your own scripts.
You’ll also see that TOOL function definitions look a lot like their C counterpart. The only noticeable difference is the lack of a declaration for the type of the parameters passed to any function as well as for the return type of the function. This “looseness” in variable types in TOOL is one of the reasons that I put in all the RTTI type functions into the engine. Another consequence of this type of undeclared variable type is that any variable can take on a value of any type. In the current implementation of TOOL, I foresaw that this could lead to memory leaks, so I attempted to address this situation in the VMVariant copy constructor, where if a variable already contains any heap allocations, it will delete them prior to adopting the new value.
Important note: Even though TOOL is a weakly typed language from the script writer's perspective, the TOOL interpreter will type-check the type of a variable passed into its intrinsic routines. If the variable is not of the right type, the interpreter will halt execution of the script and will throw an exception.
Another key difference between TOOL and C programs is that “local variables” in TOOL appear in a function's parameter list. Local variable declarations appear in a comma-separated list after a semi-colon in the function declaration. Temporary variables that are not scoped in this fashion are promoted to the global namespace. While TOOL functions will work with variables in the global namespace just fine, it can lead to some “strange and unexpected” behaviors (as in: “how did that value get there?”) during script execution. While temporary variable declarations are not required in TOOL, adopting this approach is a useful one.
As in C++, new objects of a TOOL class are initialized using a constructor function, which has the same name as the class itself. Again, refer to the CoreClasses.Tool for several examples of how classes are declared in TOOL. The last operation in a TOOL class constructor is that it returns the new object through “return( this )”. For those of you not familiar with C++, the variable this refers to the object for which the member function is being called. It is an implicit parameter passed to every non-static member function. In this case, this is the new object just created.
In TOOL classes, all data members are implicitly protected. The only way to access or modify the value of a member variable is through a member function. If you need to access a member variable outside a member function, you must provide access to member functions to do this. Again see the CoreClasses.Tool file for several examples. The new operator creates a new object of the class whose name follows it. The expressions in parentheses after the class name are the arguments to be passed to the constructor function.
TOOL also allows one class to be derived from another. The derived class will inherit the behavior of the base class and possibly add some behavior of its own. TOOL only supports single inheritance; therefore, each class can have at most one base class. The classes in the core classes file all express this attribute of the language. You can also see in TOOL how derived class constructors can call directly into their parent’s constructor. This is because the entire interface of the parent class is available to the derived class.
TOOL Script File Layout
It is possible that the TOOL Script Layout can be designed so that the run-time engine can perform a simple introspection on the TOOL Script Program to determine the nature of the program. An empty script program is shown below. This approach provides an "object interface approach" between the TOOL Runtime Engine and the scripts it runs.
In effect, the Runtime will "send a message" to the script for each stage of task operation. The procedural equivalent to this concept, is that Runtime will invoke up to five different top-level-functions in the script program; each call representing a specific stage in the completion of the task. See the sample program below for an example of an introspective approach to TOOL Scripts. However, you can design any form of similar methodology that suits your application best; since the TOOL Engine is easily driven by the host environment, it is possible for you to define even more specific file layouts that suit your particular needs.
#const FILE_ACCESS_MODE_WRITE 1
#const FILE_SHARE_READ 1
#const FILE_OPEN_ALWAYS 3
#const FILE_POINTER_REF_END 2
GetScriptInterface()
{
SetAppEnvField( "InitFunction", "Init" );
SetAppEnvField( "PreRunFunction", "PreJob" );
SetAppEnvField( "RunJobFunction", "RunJob" );
SetAppEnvField( "PostRunFunction", "PostJob" );
SetAppEnvField( "ExitFunction", "Exit" );
}
MakeLogRecord( sLogText ;dtNow, hFile, sOutput )
{
dtNow = GetDate();
sOutput = StringFormat( "[ % ]--> % \r\n",
DateToString( "%a, %b %d, %Y %X", dtNow ),
sLogText );
Echo( sOutput );
if ( !PathExists( "c:\\tool.task.log" ) )
{
CreateDirectory( "c:\\tool.task.log" );
}
hFile = OpenCreateFile( "c:\\tool.task.log\\all.tasks.log",
FILE_ACCESS_MODE_WRITE,
FILE_SHARE_READ,
FILE_OPEN_ALWAYS );
if ( 0 != hFile )
{
SetFilePointer( hFile, 0, FILE_POINTER_REF_END );
WriteLineToFile( hFile, sOutput );
CloseFile( hFile );
}
}
Init()
{
MakeLogRecord( "Task -- Init() Called." );
}
PreJob()
{
MakeLogRecord( "Task -- PreJob() Called." );
}
RunJob()
{
MakeLogRecord( "Task -- RunJob() Called." );
}
PostJob()
{
MakeLogRecord( "Task -- PostJob() Called." );
}
Exit()
{
MakeLogRecord( "Task -- Exit() Called." );
}
TOOL Script Quick Exits
If a TOOL script encounters a condition under which it can not (or should not) continue execution, then the TOOL script can make a call to the Throw API which will cause an immediate halt of the TOOL program. This API accepts a message string that can be passed back for error reporting purposes. An example of this call is shown below:
Throw( "Goodbye Cruel World" );
Getting into TOOL
I’ve covered a lot of ground here in attempting to convey how I’ve taken David Betz’s BOB into new territory. My goal for TOOL was to extend the original engine and to make it “drop in simple” to use. In this article, I’ve attempted to explain enough about the way the project is organized. I also hope that I provided enough information as to how one would extend the capabilities of the TOOL engine for their own specific capabilities and/or requirements to the project. Finally, I’ve provided a rather complete set of TOOL classes that illustrate the full range of the TOOL API and have provided some “general classes” that can be used in any TOOL script that you develop.
I have many more plans for my continuing evolution of TOOL. There are some items that really need to be addressed, such as the memory management and garbage collection issues I introduced earlier. Further, I’ve taken some steps towards having the ability to attach a debugger to the TOOL engine, and have plans to continue this pursuit as well. My to do list also includes adding many more intrinsic functions to the engine, adding multi-threaded script capabilities, integrating it with a meta-forms language I’ve also developed, etc. etc. So, I guess I’m not yet done growing TOOL. Even so, I thought that some of you folks out there would enjoy another interpreter project to add to your tool-chests.
While TOOL may be a little late to the party since there are so many other interpreter engines out there already, I think that the ease of integration and extension of the TOOL engine gives it a bit of an edge over other interpreters. And while I fully recognize that it is not as powerful or as complete as many other script engines as well, the fact that I can include it completely in my own programs and can extend it any way I want to with relative ease, makes it an odds on favorite for many of my projects.
Plus, I just really groove to the whole concept of programmable programs, and have a tool like the TOOL engine in my toolbox gives me a certain amount of jazz from the sizzle factor alone. Anyhow, I hope that some of you will also find TOOL worth catching on to, and can enjoy dabbling in an engine completely under your control to the same extent that I have.
But Wait…There is More….Bonus Material Included in TOOL
Among the other noteworthy and powerful features built into TOOL is a complete Runtime Forms Engine, that I call XMLForms. This puts a "runtime interpreted form engine" right into the interpreter and allows the creation of TOOL-applications that come packed with their own GUI.
This feature itself is worth significant study and I will now try to explain this feature from a very high level. Again, since this project is so large it is impossible for me to describe all aspects of the project here (me thinks I'd almost need a book for that level of explanation).
The entire code base for XML-Forms is in the GUI.Tools/XMLForms directory included in the distribution. There is even a small sample application in that directory just for testing XML Form scripts.
Just like the TOOL project, there is a single façade class that encapsulates the entire XML form system of classes. The definition of this class is found in the file XMLFormFacade.h and the implementation can be found in XMLFormFacade.cpp.
Creation of an XML form consists of two main operations: parsing an XML form definition, and then running a window that is built from that definition. You can see how simply this can be done by studying the XmlFormTest project which is a small utility I have used on countless occasions for testing my own XML forms in my projects. Specifically, see the OnButtonTestIt() function in the sample in the XMLFormTestDlg.cpp.
Generally speaking, the architecture of XML Forms uses an XML parser to read the form definition from a file (or a string buffer). As the XML Form data is read, a series of screen-control meta-objects is created and stored in collections in the XMLFormFacade object. When it is time for the form to be displayed, the collection of control meta-objects is given to a control factory that creates the screen controls and places them as child controls in a container window. Additional parsing services are provided in the XMLStyleDecoder object that allows the style of the controls and the containing window to be explicitly defined in the XMLForm data.
Other objects support data exchange and validation, the rules of which are also defined in the XML data string for the form. The task of data validation is contained in the DDV Wrapper object (implemented in XMLFormDDVWrapper.cpp). Data exchange services are contained in the DDX Wrapper object (implemented in XMLFomrDDXWrapper.cpp).
Tab-control management is one of the very specialized functions implemented in the XMLForms project. Normally, tab order is defined as a function of Z-Order, but this does not work well with a project like this one. So, tab-orders can be defined in the XMLForm data file and that data is processed in the XMLFormTabSetManager object. In order to assist with this task, all XMLForm controls will participate in focus (OnGainFocus() and OnLoseFocus()) notification to the containing window. You can see examples of these specialized tasks in the implementation of most of the XMLForm controls in the XMLFomrControls.cpp file.
Rendering Data
As an added feature, the background of XMLForms can be driven by another feature included in the XMLForms project, in the way that this project is integrated with the TOOL interpreter. If you see the file GiftCardPurchase.form in the samples directory, you can see examples of how TOOL script is embedded in the XMLForm data file itself. Pay particular note to the PAGE_BACKGROUND section of this sample file. If this node is present in the XMLForm file, then the XMLForm system will run the render script contained in that XML-node to perform the background paint of the XMLForm. Mixing render scripts with the XMLForm windows can create some very interesting possibilities for windows that emulate the appearance of ‘real-world’ business-forms. Although, I have to admit that creating and debugging these screens is bit of a chore to pull off by hand.
Samples Included
I have included the following samples with this initial delivery:
- ToolDll project: This project packages the TOOL Interpreter into a DLL.
- ToolDllTester project: This project utilizes the TOOL DLL and provides a sample of how to integrate with the TOOL DLL.
- ToolTester project: This project provides a sample of how to incorporate TOOL directly into your applications.
- ToolExtender project; This project is a DLL Shell project for a TOOL Extender DLL that can be bound to and executed at runtime.
- XMLFormTest project: This project provides a sample of how to include XMLForm (and TOOL) directly in your applications.
- Sample Scripts: are in the Tools folder.
- Sample XMLForms: are in the XMLForms folder. Among the most interesting screens defined here is the PayByCreditCard.form file as it is about the most complex XMLForm yet developed and shows a lot of the advanced features of this library including advanced-event-handling defined for the form (see the
ACTIONS blocks in the file). Another interesting sample of an XMLForm is the GiftCardPurchase.form in the Tools directory. This sample shows custom background rendering of an XML Form to resemble an 'actual paper form'.
- Sample Report Generation: is in the Sample.Report folder (this shows how to format a TOOL Template File (a specialized form of TOOL script)) and illustrates how to gather data from an ODBC data source and load that data into an SQLite database for intermediate storage and/or data manipulation.
How do I Use the RunTool Program
What is the RunTool Program for?
The RunTool program is supplied so that the user can run TOOL Scripts and interactive TOOL programs using the TOOL runtime. RunTool allows small "batch files" to be developed in the TOOL language for any particular use.
The RunTool program provides a stand-alone runtime environment for "desktop applications" written in TOOL. The nature and scope of these desktop applications are not limited in any way; meaning that anything that is supported in TOOL can be made a part of a desktop utility written in TOOL.
How is it Used?
Simply start the RunTool program.
If any run-time parameters should be passed to the TOOL program (for any initialization or control purposes) then enter those in the "Parameters" edit box using the following format:
PARAM=SomeParam=SomeValue PARAM=SomeOtherParam=SomeOtherValue
Note: The length of the parameters string is currently limited to 255 characters.
At this point, simply click on the "Select Script" button which will open a file dialog. Select the TOOL Script Program that you desire to run using the file dialog. Any messages produced from the TOOL Script will be shown on the RunTool program. This display can be cleared by clicking on the "Clear Messages" button.
How do I Use the ToolForge Program
What is the ToolForge Program for?
The TOOLForge program is the IDE for TOOL. As such it can be used as a syntax-aware colorizing editor for TOOL Scripts. It can also be used as an interactive debugger for TOOL Scripts. This version of TOOLForge is still a bit rough around the edges, as it is still a work in progress; so it may crash from time to time.
How is it Used?
To edit TOOL scripts, simply start the TOOLForge program and use it like you would any other editor. Here are the steps one would use to debug TOOL Scripts in the Forge:
- First notice the Toolbar in the program. On the right most side of the toolbar are three controls that drive the debugging session. The checkbox on the right is used to control the single step operation of the debugger. You should leave this checked initially.
- Next, open the 'Debug' Menu and select the debug script command. This will display a file selection dialog for you to select the script file that you would like to debug. For your first use of the debugger, you may want to use the FileBuilder.tool file included in the samples directory.
- After selecting a file, the debugger will show the current execution point of the script with a green bar through the code window.
- To toggle a break-point, click on a line of code in the code window; open the edit menu and select the toggle-break point command. Breakpoints are highlighted with a dark-blue bar through the text window, and a 'stop sign' in the gutter of the code window.
- When the single step check box is checked, clicking on the the 'blue down arrow code page' on the toolbar will execute a single step in the script. When this check box is not checked, then the script will run in 'animation mode' which means that the editor window will follow the program execution. To interrupt a script running in animation mode, simply check the single-step control to issue a program break.
- The three windows along the bottom of the program are (from left to right): Script Output Window, Call Stack Window, and Variables Display Window. Note that not all TOOL variables are completely expanded in the variable window. This is one of the items that are still in progress on this project. But, some of them work, and you'll get an idea of where I'm going with this feature.
- The call stack and variables windows are updated at each breakpoint. The script output window shows whatever was Echo'ed from the script as it runs.
- To run a script to completion, click the 'red down arrow code page' button on the toolbar.
Known Problems
Instantiating multiple XML Forms using the TOOL DLL causes the application message pump to mess up. The cause of this is not yet known, and the behavior does not manifest when XMLForms is used directly in the application.
There are sure to be many other little bugs here and there. While the project has been tested reasonably completely due to its being included in many of my projects, there is still a great likelihood that there are uncovered bugs in a project of this size.
If you find some behavior that you think may be a bug, then please do let me know. If the behavior really does prove to be a flaw, I will strive to correct it.
Help Wanted and Feature Requests
TOOL and XML Forms are the core of a pretty sizeable project. And there is still a lot of room for improvements and enhancements. Among the items on the to-do list are:
- Add additional DSN creation options. Currently only DSN creation strings for SQL Server and Access are defined.
- Enhancements to TOOL to interact with (create entries in) the Start button menus and shell functions for creating shortcuts.
- Better memory management in the TOOL RunTime including better garbage collection while the interpreter is running.
- New features that fit your specific requirements that are not addressed in this release.
In short, I recognize that while TOOL is already a feature rich interpreter, there are many more places it can grow into. That is one of the reasons why I am releasing TOOL to the community at large; it is my hope that with other folks trying to use TOOL for their own purposes, the library itself will grow in breadth. In addition, I also realize that there are some very bright folks out there who may be willing to help me find and fix the little problems that are sure to exist in the code.
A Note on the Source Code
I am an avid net crawler and always on the look out for new and interesting code I can add to TOOL (as well as other projects). And there is a lot of work in this project that has arrived from all over the place. However, I don’t usually just adopt code as is, and instead I make a standard practice to heavily reformat all ‘adopted code’. Included in these reformat exercises are the following tasks:
- Class variable renaming: converted to Hungarian notation and also made consistent with my own naming conventions.
- Header File / Implementation File headers and footers.
- Function headers and footers.
- Removal / Replacement of Tabs with spaces.
Why do I do this? Well, for the following reasons:
- Code Review: Since I will often end up extending these files, or in some cases repairing them, I find that going through these files with a ‘fine tooth comb’ provides me with additional familiarity with the organization of the files, so that if I do need to work on them, I have some idea of how they are put together.
- Consistency: It is probably a subjective and/or aesthetic choice on my part, but it is my opinion that the ‘internal implementation’ of any project should strive to be as ‘internally consistent’ as possible. In my opinion, this approach makes the overall more readily understood by others trying to get familiar with the project. In other words, I don’t think it helps if a project is created from a bunch of individuals' work where each file has its own stylistic approach to code creation. There are exceptions to this rule, for example, if I include a major open source library like SQLite or zlib, I don’t go through all their files with this level of attention, but any wrapper object I use to access those libraries will use my preferred style.
So, while it will do little good to ‘apologize’ to any other developer whose files have been modified in this manner, at least you have an explanation as to the rationale behind these changes. I try to retain the original author’s comments in any file that I integrate into TOOL and put through the edit-press as I have described, but as this project has been in motion for a really long time, there are cases where I do not even recall if some of the source is based on some project/sample that I stumbled on in my travels. If you think that any file in this project is a derivative of one of your projects and you think that you deserve credit for any part of this project, then contact me and make a case.
Finally, careful review of this code will uncover a few instances of conditional compilation wrapped in "ifndef TOOL_PUBLIC_BUILD" which has the end effect of 'removing some functionality' from this release. The reason behind these conditional compiled areas is that some functionality relied on commercial libraries that I make use of in my "private build" of the product, but that I do not have the right to release the source for. If anyone wants to come up with non-commercial-library-based replacements for these functional areas, then I would welcome that, as it would increase the utility of this project to its 'private build' levels.
Even if you don't want to use TOOL or XMLForms, you may still want to take a good long look at this pile of code as there are some classes and techniques implemented here that can find use in other ways.
A Note on this Document
TOOL continues to evolve constantly. And because of that evolution, this document may not be entirely complete with regards to the latest/newest features I've added to the API. So, if there is some area of the API not covered in this document, then hopefully the samples provided will provide some additional information.
What is Coming Next for TOOL?
There are countless thousands of new features that could be added to TOOL that I can not even begin to imagine. So, I'll leave those directional decisions up to you all. Also, I have already developed other products that make use of TOOL, such as: report generators, installers, application servers, etc. Pending interest in any one of these areas, I may also release these products in one form or another.
Fare Thee Well
If you have gotten this far in reading this article, then thanks for putting up with reading this (at times rambling) submission. I hope that you find some use for this project. And if you have any ideas for where TOOL may find use, or if you have some specific request that you'd like to see, then please contact me and perhaps I can be of some kind of assistance.
Update History
- 2006/10/16: Gosh, how the time does fly. It is hard to believe that it has already been a year since this project was updated. As evidenced by 2b|!2b==? prompting me to get off the stick. So, here we are at last with an update. This release includes fixes, updates, and enhancements (too many to recall, but the key ones follow):
- With great assistance from 2B|!2b==? (who did a large part of the "porting" work), the TOOL projects now have been updated for newer Microsoft compile environments.
- The parsing error (reported by yiqianfeng ) has been corrected.
- The TOOL engine now has built in support for both SQLite 2.x and 3.x database formats. Along with this, several new API calls have been added for SQLite 3.x management. Since this integration took place, the old "minidb" calls have been renamed (I apologize in advance if your scripts have been broken by this renaming action) since they no longer made much sense. All "minidb" calls and have been replaced with "SqlLite2" calls.
- The ODBC class "dynamic binding" behavior has been revised/improved to allow better (more portable) support for more SQL data types.
- The TOOLForge application has been enhanced to allow visual creation and testing of XMLForms. This is made possible through use of heavily modified version of very excellent screen designer classes available here by Johan Rosengren.
- The TOOL Calculator object has been enhanced to include several new built in functions:
sin, cos, exp, sqrt, ln, tan, ctg, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, atanh, log10, log2, abs, rounddown, and roundup.
- 2005/09/23: This release includes fixes, updates, edits to all problems that I was made aware of to date. As follows:
- Factorial Example error (reported by Abf23) has been corrected. The example was also proven correct with the inclusion of a sample script.
- Build errors (reported by Abf23) for bad stdafx.h path. The offending files have been corrected.
- The updated delivery includes compiled sample DLL files and executables (in response to Manuele Turini).
- The errors reported by hero3blade are not entirely addressed in this release, due to the fact that when I try to change the code to address his reports, I can no longer build the release. There could to be some as yet undetermined platform issue at play here.
