Tokenizer Wordlist


The following words are in the TOKENIZER wordlist. In Tokenizing mode, the interpreter searches this wordlist first. The interpreter executes the word if it's an immediate word or if the interpreter is not in compile mode.

Most of these words are state smart words. In other words, they have different behaviors depending on whether the interpreter is in compile mode or interpret mode. Their definitions are in TTOKEN.G.

The search order during building generally goes something like TOKENIZER ... CORE ... HOME. The HOME vocabulary is a catch-all for words that execute on the host PC. If you define a target word that has the same name as a word in the HOME wordlist, that word will mask the home wordlist's version. If this causes a problem, you can place the desired HOME wordlist words within {{ }} brackets.


Defining Words:

Most defining words have three different behaviors:

: ( <name> -- )
code ( <name> -- magic )
code{ ( <name> -- f )
VARIABLE ( <name> -- )
MARKER ( <name> -- )
2VARIABLE ( <name> -- )
CONSTANT ( <name> -- )
2CONSTANT ( <name> -- )
ARRAY ( <name> -- )
STRING ( <name> -- )
CREATE ( <name> -- )
VALUE ( <name> -- )

Other defining words:

PROGRAM ( <name> -- ) Marks the beginning of a tokenized program with a 7-byte header. Lays down 0x07C9, a 2-byte checksum and 3-byte length. Checksum and length are resolved by END.
END ( -- ) Marks the end of this block of data and resolve the checksum and length.
:: ( <name> -- ) Begins re-definition of an existing word keeping the old xt value.
code ( <name> -- magic ) Begins a code definition using the current CPU type. Sets search order to ASSEM.
END*CODE ( magic -- ) Ends a code definition, resolves the length of the code string.
Usage: CODE FOO ... C; END*CODE"
code{ ( <name> -- f ) Begins a machine language definition.
Sample usage: CODE{ FOO 1 hex[ 01 F3 DE 22 ] }CODE \ 8051
}code ( f -- ) Ends a CODE{ sequence.
%% ( <name> -- ) Declares a new token, lays the token#

Other Compilation:

," ( string" -- )
; ( -- )
C" ( <string> -- )
S" ( <string> -- )
." ( <string> -- )
[COMPILE] ( <name> -- )
POSTPONE ( <name> -- )
TO ( x <name> -- )

Conditional Compilation:

#IF ( -- ) Lays #IF# followed by a 16-bit displacement.
#ELSE ( -- ) Resolves #IF's displacement, lays 0 #IF# followed by a 16-bit displacement.
#ENDIF ( -- ) Resolves #IF's or #ELSE's displacement.
#THEN ( -- ) Same as #ENDIF.

Control Structures:

The control structure words are only here for error checking. They are: BEGIN UNTIL AGAIN WHILE REPEAT MULTI. In addition, DO ?DO LOOP and +LOOP make corrections for local variable indexing


Other:

H# ( <hex_number> -- ) OFW word to represent a hex number. Split for target's cell width if it's a double.
CHAR ( <c> -- ) Lays CHAR followed by the character <c>.
[CHAR] ( <c> -- ) Lays a literal whose value is the character <c>.
. ( -- ) Lays down (.) TYPE. token pair.
U. ( -- ) Lays down (U.) TYPE. token pair.
D. ( -- ) Lays down (D.) TYPE. token pair.
UD. ( -- ) Lays down (UD.) TYPE. token pair.
.R ( -- ) Lays down (.R) TYPE. token pair.
U.R ( -- ) Lays down (U.R) TYPE. token pair.
{ ( -- ) Begin a locals structure.