Smalltalk/X Webserver

Documentation of class 'Scanner':

Class: Scanner

Inheritance
Description
Related information
Class protocol
Instance protocol
Private classes

Inheritance:

   Object
   |
   +--Scanner
      |
      +--JavaScriptScanner
      |
      +--Parser
      |
      +--SunRPC::XDRScanner

Package:: stx:libcomp

Category:: System-Compiler

Version:: rev: 1.254 date: 2010/02/08 12:03:10; user: cg; file: Scanner.st directory: libcomp; module: stx stc-classLibrary: libcomp

Author:: Claus Gittinger

Description:

Scanner reads from a stream and returns individual smalltalk tokens
Its main method is #nextToken, which reads and returns the next token
Possibly placing additional information (such as tokenValue) into 
instance variables as a side effect.

TODO: 
    some testers noticed, that ST-80's scanner methods are called
    xLetter, xDigit etc. For code using these (internals), the nextNumber,
    nextIdentifier etc. methods should be renamed.
    (to me, these seem to be internal private methods; 
     their public use is not a good idea ..)

    Scanner is typically subclassed for parsing and #nextToken
    invoked via #self-sends.
    This should be changed and scanner ought to be an instance variable
    of Parser - this allows more flexible use of the scanner/parser
    framework (i.e. changing the scanner without affecting the parser).

Extensions:
    this scanner allows for 3-character binary selectors.
    also, # is a valid selector. (however, ## is currently scanned as a symbol literal).

Related information:

    Parser

Class protocol:

Signal constants

doNotShowCompilerWarningAgainActionQuery
emptySourceNotificationSignal

accessing

actionArray
flushActionArray
typeArray

class initialization

initialize: initialize the classes defaults. Typically, these are changed
later in the 'private.rc' file.

defaults

allowDollarInIdentifier: return true, if $-characters are allowed in identifiers.
Notice, that dollars are NEVER allowed as the first character in an identifier.
allowDollarInIdentifier: aBoolean: this allows turning on/off $-characters in identifiers.
Notice, that dollars are NEVER allowed as the first character in an identifier.
If turned off (the default), dollars are not allowed in identifiers,
but instead are scanned as character-constant prefix.
If turned on, dollars are in identifiers are allowed, while extra
dollars are still scanned as constant character prefix.
If you have to fileIn old VW-Vsn2.x classes, turn this off
before filing them in; i.e.:
Compiler allowDollarInIdentifiers:false
allowDolphinExtensions: return true, if ##(..) computed literal Arrays are allowed
allowDolphinExtensions: aBoolean: this allows turning on/off support for computed literal Arrays ##(..) as in dolphin.
If you want to fileIn Dolphin classes, enable this with:
Compiler allowDolphinComputedArrays:true
allowLiteralNameSpaceSymbols: return true, if literal nameSpace symbols are allowed (#foo::bar) are allowed
allowOldStyleAssignment: return true, if underscore-assignment (pre ST-80v4 syntax) are to be allowed
allowOldStyleAssignment: aBoolean: this allows turning on/off recognition of underscore-assignment (pre ST-80v4 syntax).
You must turn this off, if code with variables named '_' is to be filedIn
allowQualifiedNames: return true, if #{..} qualified names are allowed
allowQualifiedNames: aBoolean: this allows turning on/off support for qualifiedNames #{ .., } as in vw3.
If you want to fileIn vw3 or later classes, enable this with:
Compiler allowQualifiedNames:true
Notice, that qualified names are not really supported semantically
(they are parsed, but treated like regular globals)
allowSqueakExtensions: return true, if support for squeak extensions
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)
is enabled.
allowSqueakExtensions: aBoolean: this allows turning on/off support for squeak extensions:
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)

If you want to fileIn Squeak classes, enable this with:
Compiler allowSqueakComputedArrays:true
allowUnderscoreInIdentifier: return true, if underscores are allowed in identifiers
allowUnderscoreInIdentifier: aBoolean: this allows turning on/off underscores in identifiers.
If turned off (the default), underscores are not allowed in identifiers,
but instead scanned as assignment character (old ST/80 syntax).
If turned on, underscores are in identifiers are allowed, while extra
underscores are still scanned as assignment.
If you have to fileIn old VW-Vsn2.x classes,
turn them off with:
Compiler allowUnderscoreInIdentifiers:false
maxBinarySelectorSize
warnCommonMistakes: return true, if common beginners mistakes are to be warned about
warnCommonMistakes: aBoolean: this allows turning on/off warnings about common beginners mistakes.
Those are not really errors in the strict sense, but often lead to
run time errors later.
Examples are: expr or:expr2, where expr2 is not a block.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnCommonMistakes:false
in your 'private.rc' file
warnDollarInIdentifier: return true, if $-characters in identifiers are to be warned about
warnDollarInIdentifier: aBoolean: this allows turning on/off warnings about $-characters in identifiers.
You may find those warnings useful, to make certain that your code
is portable to other smalltalk versions, which do not allow this
(i.e. VW releases 2.x and maybe others).
Notice, that dollars are NEVER allowed as the first character in an identifier.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnDollarInIdentifier:false
in your 'private.rc' file
warnOldStyleAssignment: return true, if underscore-assignment (pre ST-80v4 syntax) are to be warned about
warnOldStyleAssignment: aBoolean: this allows turning on/off warnings about underscore-assignment (pre ST-80v4 syntax).
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnOldStyleAssignment:false
in your 'private.rc' file
warnPossibleIncompatibilities: return true, if possible incompatibilities (with other ST systems)
are to be warned about
warnPossibleIncompatibilities: aBoolean: this turns warnings about possible incompatibilities (with other ST systems)
on or off.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnPossibleIncompatibilities:false
in your 'private.rc' file.
warnSTXSpecials: return true, if ST/X specials are to be warned about
warnSTXSpecials: aBoolean: this allows turning on/off warnings about stx specials.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnSTXSpecials:false
in your 'private.rc' file
warnUnderscoreInIdentifier: return true, if underscores in identifiers are to be warned about
warnUnderscoreInIdentifier: aBoolean: this allows turning on/off warnings about underscores in identifiers.
You may find those warnings useful, to make certain that your code
is portable to other smalltalk versions, which do not allow this
(i.e. VW releases 2.x).
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnUnderscoreInIdentifier:false
in your 'private.rc' file
warnings: return true, if any warnings are to be shown
warnings: aBoolean: this allows turning on/off all warnings; the default is on.
You can turn off warnings in your 'private.rc' file with
Compiler warnings:false

initialization

binarySelectorCharacters: return a collection of characters which are allowed in binary selectors
extendedBinarySelectorCharacters: return a collection of characters which are optionally allowed in binary selectors
setupActions: initialize the scanners actionTables - these are used to dispatch
into scanner methods as characters are read.
Compatibility note: in previous versions, these tables used to be kept
in classVariables, which made reuse hard as subclasses had no easy way of
defining their own tables. These are now class instance variables.

instance creation

for: aStringOrStream: create & return a new scanner reading from aStringOrStream
new: create & return a new scanner

utility scanning

scanNumberFrom: aStream: utility - helper for Number>>readSmalltalkSyntaxFrom:

Instance protocol:

Compatibility-ST80

endOfLastToken: return the position of the token which was just read.
This method was required by some PD program.
It is not maintained and may be removed without notice.
scan: aStringOrStream: initialize the scanner: set the source-stream and
preread the first token
scanDoing: aBlock: scan, evaluating aBlock for every scanned token.
scanToken: read the next token from my input stream
scanTokens: aStringOrStream: return a collection of symbolic tokens from the passed input

accessing

comments
exitBlock: aBlock
inArrayLiteral: aBoolean
lineNumber
newSourceStream: aStream
outStream: aStream
saveComments: aBoolean: not yet implemented
sourcePosition
sourceStream
tokenLineNr
tokenName
tokenPosition
tokenType
tokenValue

accessing-flags

allowDollarInIdentifier
allowDollarInIdentifier: something
allowLiteralNameSpaceSymbols
allowLiteralNameSpaceSymbols: aBoolean
allowOldStyleAssignment
allowOldStyleAssignment: aBoolean
allowSqueakExtensions: return true, if support for squeak extensions
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)
is enabled.
allowSqueakExtensions: aBoolean: this allows turning on/off support for squeak extensions:
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)
allowUnderscoreInIdentifier
allowUnderscoreInIdentifier: aBoolean
parserFlags
parserFlags: aParserFlagsInstance
scanColonAsKeyword: not used here, but eases subclassing for other languages.
warnCommonMistakes
warnCommonMistakes: aBoolean
warnDollarInIdentifier
warnDollarInIdentifier: aBoolean
warnOldStyleAssignment
warnOldStyleAssignment: aBoolean
warnPossibleIncompatibilities: return true, if possible incompatibilities (with other ST systems)
are to be warned about
warnPossibleIncompatibilities: aBoolean
warnSTXNameSpaceUse
warnSTXNameSpaceUse: aBoolean
warnSTXSpecialComment
warnSTXSpecialComment: aBoolean
warnUnderscoreInIdentifier
warnUnderscoreInIdentifier: aBoolean

directives

parseClassDirective: Class: className
parseClassHintDirective: ClassHint: className
parseDirective: parse a directive inside a comment (introduced with '{').
This is an ST/X special
parseDirectiveClassNameArg: helper for parsing a directive
parseDirectiveStringArg: helper for parsing a directive
parseDirectiveStringArg: firstCharacterCheckBlock rest: restCharacterCheckBlock: helper for parsing a directive
parseDirectiveStringListArg: helper for parsing a directive
parseNamespaceDirective: Namespace: 'nameSpace'
Namespace: nameSpace
parsePackageDirective: Package: 'name-of-package'
Package: packageId
parsePrerequisitesDirective: Prerequisites: 'name-of-package1', ... , 'name-of-packageN'
parseRequiresDirective: Require: 'name-of-feature', ... , 'name-of-featureN'
parseSyntaxDirective: Syntax: 'st-syntax-id'
parseUsesDirective: Uses: 'nameSpace1', ... , 'nameSpaceN'
Uses: nameSpaceId1, ... , nameSpaceIdN

dummy-syntax highlighting

markCommentFrom: pos1 to: pos2: Created: / 31.3.1998 / 13:34:45 / cg
markConstantFrom: pos1 to: pos2: Created: / 1.4.1998 / 13:02:56 / cg
markStringFrom: pos1 to: pos2: Created: / 31.3.1998 / 16:37:18 / cg
markSymbolFrom: pos1 to: pos2: Created: / 1.4.1998 / 12:58:42 / cg

error handling

correctableError: message position: pos1 to: pos2: report an error which can be corrected by compiler -
return non-false, if correction is wanted (there is more than
true/false returned here)
correctableSelectorWarning: message position: pos1 to: pos2: report a warning which can be corrected by compiler -
return non-false, if correction is wanted (there is more than
true/false returned here)
errorMessagePrefix
ignorableParseError: message
invalidCharacter: ch
lastTokenLineNumber: return the line number of the token which was just read.
notifyError: aMessage position: position to: endPos: notify requestor of an error - if there is no requestor
put it on the transcript. Requestor is typically the CodeView
in which the accept/doIt was triggered, or the PositionableStream
which does the fileIn. The requestor may decide how to highlight the
error (and/or to abort the compile).
Return the result passed back by the requestor.
notifyWarning: aMessage doNotShowAgainAction: doNotShowAgainAction position: position to: endPos: notify requestor of an warning - if there is no requestor, just ignore it.
Return the result passed back from the requestor (or false, if there is none).
notifyWarning: aMessage position: position to: endPos: notify requestor of an warning - if there is no requestor
put it on the transcript.
Return the result passed back by the requestor.
parseError: aMessage: report an error
parseError: aMessage line: lNr: report an error
parseError: aMessage position: position: report an error
parseError: aMessage position: position to: endPos: report an error
positionFromLineNumber: lNr
showErrorMessage: aMessage position: pos: show an errormessage on the Transcript
syntaxError: aMessage: a syntax error happened - position is not known
syntaxError: aMessage position: position: a syntax error happened - only start position is known
syntaxError: aMessage position: position to: endPos: a syntax error happened
warnCommonMistake: msg at: position: warn about a common beginners mistake
warnCommonMistake: msg position: pos1 to: pos2: warn about a common beginners mistake
warnDollarAt: position: warn about $-character in an identifier
warnOldStyleAssignmentAt: position: warn about an oldStyle assignment
warnPeriodAt: position: warn about a period in an identifier
warnPossibleIncompatibility: msg position: pos1 to: pos2: warn about a possible incompatibility with other ST systems
warnSTXSpecialCommentAt: position to: endPosition
warnUnderscoreAt: position: warn about an underscore in an identifier
warning: aMessage: a warning - position is not known
warning: aMessage doNotShowAgainAction: doNotShowAgainAction position: position to: endPos: a warning
warning: aMessage line: lNr: a warning - only start position is known
warning: aMessage position: position: a warning - only start position is known
warning: aMessage position: position to: endPos: a warning
warningMessagePrefix

general scanning

scanNumberFrom: aStringOrStream: scan aSourceString for the next number in smalltalk syntax
Return the number or nil.
scanPositionsFor: aTokenString inString: aSourceString: scan aSourceString for occurrances of aTokenString.
Return a collection of start positions.
Added for VW compatibility (to support simple syntax-highlight).

initialization

initialize: initialize the scanner
initializeActionTable
initializeFlagsFrom: aScanner: initialize flags from another scanner
initializeFor: aStringOrStream: initialize the new scanner & prepare for reading from aStringOrStream
requestor: anObject: set the requestor to be notified
setSource: newSource
source: aStringOrStream: prepare for reading from aStringOrStream

parser interface

errorFlag: flagArg
token
tokenLineNr: lineNumberArg
tokenPosition: positionArg

private

backupPosition: if reading from a stream, at the end we might have read
one token too many
beginComment
checkForKeyword: string: check if string is a keyword (as opposed to an identifier).
That is, its one of 'self', 'super', 'nil', 'true', 'false',
or 'thisContext'.
'here' is handled elsewhere (since it must be treated as an
identifier, if declared locally.
collectedSource
eatPeekChar
endComment: comment
endComment: commentString type: commentType
escapeCharacterFor: aCharacter: only if AllowExtendedSTXSyntax is true
For now: do not use, since stc does not support it.

much like character escapes in C-literals;
expands:
\n newLine
\r return
\t tab
\b backspace
\f formfeed
\g bell

\\ backSlash
\ ...\ (backslash-separator) ignored up to next backslash
\xNN hexCharacter
\uNNNN hex UnicodeCharacter
ignoreErrors: return the flag which controls notification of errors
ignoreErrors: aBoolean: enable/disable notification of errors
ignoreWarnings: return the flag which controls notification of warnings
ignoreWarnings: aBoolean: enable/disable notification of warnings
isCommentCharacter: ch: return true, if ch is the comment-start character.
Brought into a separate method to allow for easier subclassing
isSpecialOrExtendedSpecialCharacter: ch
notifying: anObject: set the requestor to be notified
setNameSpace: aNameSpace: ignored here
setPackage: aNameSpace: ignored here
setSyntax: aSyntax

reading next token

nextCharacter: a $ has been read - return a character token
nextColonOrAssign: colon has been read - look for = to make it an assign
nextExcla: a ! has been read - return either
the ! binarySelector,
ExclaLeftParen (for '!('),
ExclaLeftBrack (for '!['),
ExclaLeftBrace (for '!{')
nextExtendedSpecial: ch
nextHash: a # has been read - return either
a symbol,
HashLeftParen (for '#('),
HashLeftBrack (for '#['),
HashLeftBrace (for '#{' and AllowQualifiedNames)
HashHashLeftParen (for '##(' and AllowDolphinExtensions)
HashHashLeftBrack (for '##[' )
HashHash (for '##' )
nextId: no longer used here - remains for backwardCompatibility for
subclass users ... (sigh)
nextIdentifier: an alpha character (or underscore if AllowUnderscore) has been read.
Return the next identifier.
nextMantissa: radix: read the mantissa of a radix number

** This is an obsolete interface - do not use it (it may vanish in future versions) **
nextMantissaAndScaledPartWithRadix: radix: read the mantissa of a radix number.
Since we dont know yet if this is for a Float, LongFloat or a FixedPoint,
return info which is useful for all: an array consisting of
the mantissa as float/longFloat, the numerator and the scale for a fixedPoint.
nextNumber: scan a number; handles radix prefix, mantissa and exponent.
Allows for e, d or q to be used as exponent limiter.
nextPrimitive: scan an inline C-primitive.
nextSpecial: a special character has been read, look for another one.
also -number is handled here
nextSpecialWith: firstChar: a special character has been read, look for another one.
also -number is handled here
nextString: delimiter: a single quote has been scanned; scan the string (caring for doubled quotes
nextSymbolAfterHash: helper: a # has been read - return #Symbol token or nil
nextToken: return the next token from the source-stream
nextToken: aCharacter: return a character token
nextUnderline: return a character token
skipComment: skip over a comment; handles ST/X eol comments.
skipToEndOfLineRememberingIn: commentStreamOrNil

Private classes:

    Comment
    Directive
    DoNotShowCompilerWarningAgainActionQuery

ST/X 6.1.1; WebServer 1.620 at exept:8081; Tue, 22 May 2012 22:00:03 GMT