Smalltalk/X Webserver

Documentation of class 'Scanner':

Class: Scanner

Inheritance
Description
Class protocol
Instance protocol
Private classes

Inheritance:

   Object
   |
   +--Scanner
      |
      +--JavaScriptScanner
      |
      +--Parser
      |
      +--SunRPC::XDRScanner

Package:: stx:libcomp

Category:: System-Compiler

Version:: rev: 1.465 date: 2024/04/22 16:57:54; user: stefan; file: Scanner.st directory: libcomp; module: stx stc-classLibrary: libcomp

Description:

Scanner reads from a stream and returns individual smalltalk tokens
Its main method is #nextToken, which reads and returns the next token
Possibly placing additional information (such as tokenValue) into
instance variables as a side effect.

TODO:
    some testers noticed that ST-80's scanner methods are called
    xLetter, xDigit etc. For code using these (internals), the nextNumber,
    nextIdentifier etc. methods should be renamed.
    (to me, these seem to be internal private methods;
     their public use is not a good idea ..)

    Scanner is typically subclassed for parsing and #nextToken
    invoked via #self-sends.
    This should be changed and scanner ought to be an instance variable
    of Parser - this allows more flexible use of the scanner/parser
    framework (i.e. changing the scanner without affecting the parser).

Extensions:
    selectors:
        this scanner allows for 3-character binary selectors.
        also, # is a valid selector. (however, ## is currently scanned as a symbol literal).

    special strings:
        c'...' a string with c escapes
        e'...' a string with c escapes and embedded expressions
        i'...' a string with c escapes and embedded expressions and internationalization
        r'...' a regex

copyrightCOPYRIGHT (c) 1989 by Claus Gittinger
             All Rights Reserved

This software is furnished under a license and may be used
only in accordance with the terms of that license and with the
inclusion of the above copyright notice.   This software may not
be provided or otherwise made available to, or used by, any
other person.  No title to or ownership of the software is
hereby transferred.

bugsarray constant containing keywords as in:
     #(
             foo:bar:
             fee:baz:
      )

is scanned as 4-element array containing ( #foo: #bar: #fee: #baz: )
this MUST be fixed.

workaround:
     #(
             #'foo:bar:'
             #'fee:baz:'
      )

return true, if $-characters are allowed in identifiers.
Notice, that dollars are NEVER allowed as the first character in an identifier.

** This is an obsolete interface - do not use it (it may vanish in future versions) **

allowDollarInIdentifier: aBoolean

this allows turning on/off $-characters in identifiers.
Notice, that dollars are NEVER allowed as the first character in an identifier.
If turned off (the default), dollars are not allowed in identifiers,
but instead are scanned as character-constant prefix.
If turned on, dollars are in identifiers are allowed, while extra
dollars are still scanned as constant character prefix.
If you have to fileIn old VW-Vsn2.x classes, turn this off
before filing them in; i.e.:
Compiler allowDollarInIdentifiers:false

allowDolphinExtensions

return true, if ##(..) computed literal Arrays are allowed

allowDolphinExtensions: aBoolean

this allows turning on/off support for computed literal Arrays ##(..) as in dolphin.
If you want to fileIn Dolphin classes, enable this with:
Compiler allowDolphinComputedArrays:true

Usage example(s):

     self allowDolphinExtensions:true
     self allowDolphinExtensions:false

allowLiteralNameSpaceSymbols

return true, if literal nameSpace symbols are allowed (#foo::bar) are allowed

** This is an obsolete interface - do not use it (it may vanish in future versions) **

allowOldStyleAssignment

return true, if underscore-assignment (pre ST-80v4 syntax) are to be allowed

allowOldStyleAssignment: aBoolean

this allows turning on/off recognition of underscore-assignment (pre ST-80v4 syntax).
You must turn this off, if code with variables named '_' is to be filedIn

allowQualifiedNames

return true, if '#{..}' and 'namespace.varName' qualified names are allowed

** This is an obsolete interface - do not use it (it may vanish in future versions) **

allowQualifiedNames: aBoolean

this allows turning on/off support for qualifiedNames #{ .., } as in vw3.
If you want to fileIn vw3 or later classes, enable this with:
Compiler allowQualifiedNames:true
Notice, that qualified names are not really supported semantically
(they are parsed, but treated like regular globals)

Usage example(s):

     self allowQualifiedNames:true
     self allowQualifiedNames:false

allowSqueakExtensions

return true, if support for squeak extensions
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)
is enabled.

** This is an obsolete interface - do not use it (it may vanish in future versions) **

allowSqueakExtensions: aBoolean

this allows turning on/off support for squeak extensions:
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)

If you want to fileIn Squeak classes, enable this with:
Compiler allowSqueakComputedArrays:true

Usage example(s):

     self allowSqueakExtensions:true
     self allowSqueakExtensions:false

allowUnderscoreInIdentifier

return true, if underscores are allowed in identifiers

** This is an obsolete interface - do not use it (it may vanish in future versions) **

allowUnderscoreInIdentifier: aBoolean

this allows turning on/off underscores in identifiers.
If turned off (the default), underscores are not allowed in identifiers,
but instead scanned as assignment character (old ST/80 syntax).
If turned on, underscores are in identifiers are allowed, while extra
underscores are still scanned as assignment.
If you have to fileIn old VW-Vsn2.x classes,
turn them off with:
Compiler allowUnderscoreInIdentifiers:false

maxBinarySelectorSize

warnCommonMistakes

return true, if common beginners mistakes are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

warnCommonMistakes: aBoolean

this allows turning on/off warnings about common beginners mistakes.
Those are not really errors in the strict sense, but often lead to
run time errors later.
Examples are: expr or:expr2, where expr2 is not a block.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnCommonMistakes:false
in your 'private.rc' file

warnDollarInIdentifier

return true, if $-characters in identifiers are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

warnDollarInIdentifier: aBoolean

this allows turning on/off warnings about $-characters in identifiers.
You may find those warnings useful, to make certain that your code
is portable to other smalltalk versions, which do not allow this
(i.e. VW releases 2.x and maybe others).
Notice, that dollars are NEVER allowed as the first character in an identifier.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnDollarInIdentifier:false
in your 'private.rc' file

warnOldStyleAssignment

return true, if underscore-assignment (pre ST-80v4 syntax) are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

warnOldStyleAssignment: aBoolean

this allows turning on/off warnings about underscore-assignment (pre ST-80v4 syntax).
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnOldStyleAssignment:false
in your 'private.rc' file

warnPossibleIncompatibilities

return true, if possible incompatibilities (with other ST systems)
are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

warnPossibleIncompatibilities: aBoolean

this turns warnings about possible incompatibilities (with other ST systems)
on or off.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnPossibleIncompatibilities:false
in your 'private.rc' file.

warnSTXSpecials

return true, if ST/X specials are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

warnSTXSpecials: aBoolean

this allows turning on/off warnings about stx specials.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnSTXSpecials:false
in your 'private.rc' file

warnUnderscoreInIdentifier

return true, if underscores in identifiers are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

warnUnderscoreInIdentifier: aBoolean

this allows turning on/off warnings about underscores in identifiers.
You may find those warnings useful, to make certain that your code
is portable to other smalltalk versions, which do not allow this
(i.e. VW releases 2.x).
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnUnderscoreInIdentifier:false
in your 'private.rc' file

warnings

return true, if any warnings are to be shown

warnings: aBoolean

this allows turning on/off all warnings; the default is on.
You can turn off warnings in your 'private.rc' file with
Compiler warnings:false

initialization

binarySelectorCharacters

return a collection of characters which are allowed in binary selectors

extendedBinarySelectorCharacters

return a collection of characters which are optionally allowed in binary selectors

extendedSingleCharacterUnarySelectorCharacters

return a collection of characters which are optionally allowed as unary selectors

setupActions

initialize the scanner's actionTables - these are used to dispatch
into scanner methods as characters are read.
Compatibility note: in previous versions, these tables used to be kept
in classVariables, which made reuse hard as subclasses had no easy way of
defining their own tables. These are now class instance variables.

Usage example(s):

     Scanner setupActions
     Scanner withAllSubclassesDo:[:cls | cls setupActions ]

instance creation

for: aStringOrStream: create & return a new scanner reading from aStringOrStream
new: create & return a new scanner

utility scanning

scanNumberFrom: aStream

utility - helper for Number>>readSmalltalkSyntaxFrom:

Usage example(s):

     |s|

     s := '12345abcd' readStream.
     Transcript showCR:(self scanNumberFrom:s).
     Transcript showCR:(s upToEnd).

Usage example(s):

     |s|

     s := '16rffffxabcd' readStream.
     Transcript showCR:(self scanNumberFrom:s).
     Transcript showCR:(s upToEnd).

Usage example(s):

     |s|

     s := '1.2345abcd' readStream.
     Transcript showCR:(self scanNumberFrom:s).
     Transcript showCR:(s upToEnd).

Usage example(s):

     |s|

     s := '1.abcd' readStream.
     Transcript showCR:(self scanNumberFrom:s).
     Transcript showCR:(s upToEnd).

Instance protocol:

Compatibility-ST80

endOfLastToken

return the position of the token which was just read.
This method was required by some PD program.
It is not maintained and may be removed without notice.

scan: aStringOrStream

initialize the scanner: set the source-stream and
preread the first token

scanDoing: aBlock

scan, evaluating aBlock for every scanned token.

scanToken

read the next token from my input stream

scanTokens: aStringOrStream

return a collection of symbolic tokens from the passed input,
similar to what would be read when reading '#( <string> )'

Usage example(s):

     Scanner new scanTokens:'name ident #sym ''string'' 8r12'

     Scanner new scanTokens:'translate (200px 100px)'

     Scanner new
	scanTokens:'Boolean subclass:#True
				instanceVariableNames:''''
				classVariableNames:''''
				poolDictionaries:''''
				category:''Kernel-Objects''
	'

Compatibility-Squeak

notify: message at: codeStreamPosition: Squeak compatibility

accessing

atEnd: true if the scanned source stream's end has been reached
comments
exitBlock: aBlock
expectToken: expectedToken
getCollectedComments: retrieve the so far collected comments, reset comment collection

** This is an obsolete interface - do not use it (it may vanish in future versions) **
hasError: return true if there where any errors (valid after parsing)
inArrayLiteral: aBoolean
lineNumber
newSourceStream: aStream
outStream: aStream: if set, this stream gets the parsed comments as it is read.
Useful to collect comments while reading
requestor
saveComments: aBoolean
sourcePosition
sourcePositionWithoutPeeks
sourceStream
tokenEndPosition: the character position of the previously scanned token's last character
tokenLineNr
tokenName
tokenPosition: the character position of the previously scanned token
tokenStartPosition: the character position of the previously scanned token
tokenType
tokenValue

accessing-flags

allowDollarInIdentifier: ** This is an obsolete interface - do not use it (it may vanish in future versions) **
allowDollarInIdentifier: something
allowLiteralNameSpaceSymbols: ** This is an obsolete interface - do not use it (it may vanish in future versions) **
allowLiteralNameSpaceSymbols: aBoolean
allowOldStyleAssignment: ** This is an obsolete interface - do not use it (it may vanish in future versions) **
allowOldStyleAssignment: aBoolean
allowSqueakExtensions: return true, if support for squeak extensions
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)
is enabled.

** This is an obsolete interface - do not use it (it may vanish in future versions) **
allowSqueakExtensions: aBoolean: this allows turning on/off support for squeak extensions:
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)
allowUnderscoreInIdentifier: ** This is an obsolete interface - do not use it (it may vanish in future versions) **
allowUnderscoreInIdentifier: aBoolean
didWarnAboutOldStyleAssignment
parserFlags
parserFlags: aParserFlagsInstance
scanColonAsKeyword: not used here, but eases subclassing for other languages.
warnCommonMistakes: ** This is an obsolete interface - do not use it (it may vanish in future versions) **
warnCommonMistakes: aBoolean
warnDollarInIdentifier: ** This is an obsolete interface - do not use it (it may vanish in future versions) **
warnDollarInIdentifier: aBoolean
warnOldStyleAssignment: ** This is an obsolete interface - do not use it (it may vanish in future versions) **
warnOldStyleAssignment: aBoolean
warnPossibleIncompatibilities: return true, if possible incompatibilities (with other ST systems)
are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **
warnPossibleIncompatibilities: aBoolean
warnSTXNameSpaceUse
warnSTXNameSpaceUse: aBoolean
warnSTXSpecialComment
warnSTXSpecialComment: aBoolean
warnUnderscoreInIdentifier: ** This is an obsolete interface - do not use it (it may vanish in future versions) **
warnUnderscoreInIdentifier: aBoolean

directives

parseClassDirective: Class: className
parseClassHintDirective: ClassHint: className
parseDirective: parse a directive inside a comment (introduced with '{').
This is an ST/X special
parseDirectiveClassNameArg: helper for parsing a directive
parseDirectiveStringArg: helper for parsing a directive.
scans a word starting with a letter or underline,
and consisting of letters, digits, underlines or dots.
parseDirectiveStringArg: firstCharacterCheckBlock rest: restCharacterCheckBlock: helper for parsing a directive
parseDirectiveStringListArg: helper for parsing a directive
parseNamespaceDirective: Namespace: 'nameSpace'
Namespace: nameSpace
parsePackageDirective: Package: 'name-of-package'
Package: packageId
parsePrerequisitesDirective: Prerequisites: 'name-of-package1', ... , 'name-of-packageN'
parseRequiresDirective: Require: 'name-of-feature', ... , 'name-of-featureN'
parseSyntaxDirective: Syntax: 'st-syntax-id'
parseUsesDirective: Uses: 'nameSpace1', ... , 'nameSpaceN'
Uses: nameSpaceId1, ... , nameSpaceIdN

dummy-syntax highlighting

markCommentFrom: pos1 to: pos2
markConstantFrom: pos1 to: pos2: intentionally left empty
markErrorFrom: pos1 to: pos2
markStringFrom: pos1 to: pos2
markSymbolFrom: pos1 to: pos2

error handling

actionToDisableWarning: warningSelector

return an action for the do-not-show-again dialogs

Usage example(s):

     self basicNew actionToDisableWarning:#warnAboutDeadCodeAndFixMe

clearErrorFlag

correctableError: message position: pos1 to: pos2

report an error which can be corrected by compiler -
return non-false, if correction is wanted
(there is more than true/false returned here)

correctableSelectorWarning: message position: pos1 to: pos2

report a warning which can be corrected by compiler -
return non-false, if correction is wanted (there is more than
true/false returned here)

correctableWarning: aMessage doNotShowAgainAction: doNotShowAgainAction doNotShowAgainForThisMethodAction: doNotShowAgainForThisMethodAction position: position to: endPos

notify requestor of a warning - if there is no requestor, just ignore it.
Return the result passed back from the requestor (or false, if there is none).

correctableWarning: message position: pos1 to: pos2

report an error which can be corrected by compiler -
return non-false, if correction is wanted
(there is more than true/false returned here - i.e. a symbol, describing how to fix it)

disableWarningsOnCurrentMethodFor: flagName

ignored here

errorFlag: flagArg

errorMessagePrefix

ignorableParseError: message

if proceeded, install method anyway (used with non-st/x primitives)

invalidCharacter: ch

back arrow

lastTokenLineNumber

return the line number of the token which was just read.

notifyError: aMessage as: errorClassOrNil position: position to: endPos lineNr: lineNrOrNil

notify requestor of an error - if there is no requestor
put it on the transcript. Requestor is typically the CodeView
in which the accept/doIt was triggered, or the PositionableStream
which does the fileIn. The requestor may decide how to highlight the
error (and/or to abort the compile).
Return the result passed back by the requestor.

notifyError: aMessage position: position to: endPos

notifyError: aMessage position: position to: endPos lineNr: lineNrOrNil

notifyWarning: aMessage doNotShowAgainAction: doNotShowAgainAction doNotShowAgainForThisMethodAction: doNotShowAgainForThisMethodAction position: position to: endPos

notify requestor of a warning - if there is no requestor, just ignore it.
Return the result passed back from the requestor (or false, if there is none).

notifyWarning: aMessage doNotShowAgainAction: doNotShowAgainAction position: position to: endPos

notify requestor of a warning - if there is no requestor, just ignore it.
Return the result passed back from the requestor (or false, if there is none).

notifyWarning: aMessage position: position to: endPos

notify requestor of an warning - if there is no requestor
put it on the transcript.
Return the result passed back by the requestor.

parseError: aMessage

report an error

parseError: aMessage as: errorClassOrNil position: position to: endPos

report an error

parseError: aMessage line: lNr

report an error

parseError: aMessage position: position

report an error

parseError: aMessage position: position to: endPos

report an error

positionFromLineNumber: lNr

setErrorFlag

showErrorMessage: aMessage position: pos

show an errormessage on the Transcript

syntaxError: aMessage

a syntax error happened - position is not known

syntaxError: aMessage position: position

a syntax error happened - only start position is known

syntaxError: aMessage position: position to: endPos

a syntax error happened.
Return true for correction, false if not

tokenInErrorMessage

warnCommonMistake: msg at: position

warn about a common beginners mistake

warnCommonMistake: msg position: pos1 to: pos2

warn about a common beginners mistake

warnDollarAt: position

warn about $-character in an identifier

warnOldStyleAssignmentAt: position

warn about an oldStyle assignment

warnParagraphAt: position

warn about §-character in an identifier

warnPeriodAt: position

warn about a period in an identifier

warnPossibleIncompatibility: msg position: pos1 to: pos2

warn about a possible incompatibility with other ST systems

warnSTXSpecialCommentAt: position to: endPosition

only warn once

warnUnderscoreAt: position

warn about an underscore in an identifier

warning: aMessage

a warning - position is not known

warning: aMessage doNotShowAgainAction: doNotShowAgainAction doNotShowAgainForThisMethodAction: doNotShowAgainForThisMethodAction position: position to: endPos

a warning

warning: aMessage doNotShowAgainAction: doNotShowAgainAction position: position to: endPos

a warning

warning: aMessage line: lNr

a warning - only start position is known

warning: aMessage position: position

a warning - only start position is known

warning: aMessage position: position to: endPos

a warning

warningMessagePrefix

general scanning

scanNumberFrom: aStringOrStream

scan aSourceString for the next number in smalltalk syntax.
Return the number or nil. Leave the position of the stream after the number
or unchanged (if no number is coming)
Extension:
since constant fractions (int/int) are treated like lexical tokens (not messages)
in Smalltalk/X, this also reads fractions.

scanPositionsFor: aTokenString inString: aSourceString

scan aSourceString for occurrences of aTokenString.
Return a collection of start positions.
Added for VW compatibility (to support simple syntax-highlight).

Usage example(s):

     Scanner new scanPositionsFor:'hello' inString:'foo bar hello baz hello'
     Scanner new scanPositionsFor:'3.14' inString:'foo 3.145 bar hello 3.14 baz hello 3.14'
     Scanner new scanPositionsFor:'16' inString:'foo 16 bar hello 16r10 baz hello 2r10000'

initialization

initialize: initialize the scanner
initializeActionTable
initializeFlagsFrom: aScanner: initialize flags from another scanner
initializeFor: aStringOrStream: initialize the new scanner & prepare for reading from aStringOrStream
requestor: anObject: set the requestor to be notified
setSource: newSourceStream
source: aStringOrStream: prepare for reading from aStringOrStream;
notice: if token is nonNil, it is preserved. This allows for scanning
across streams.

parser interface

token
tokenLineNr: lineNumberArg
tokenPosition: positionArg

private

backupPosition: if reading from a stream, at the end we might have read
one token too many
beginComment
characterNamed: name
checkForKeyword: string: check if string is a keyword (as opposed to an identifier).
That is, its one of 'self', 'super', 'nil', 'true', 'false',
or 'thisContext'.
'here' is handled elsewhere (since it must be treated as an
identifier, if declared locally.
collectedSource
eatPeekChar
endComment: comment
endComment: commentString type: commentType: obsolete; left for backward compatibility
endComment: commentString type: commentType start: startPos end: endPos
eolIsWhiteSpace
escapeCharacterFor: aCharacter escapeStyle: escapeStyle: only if AllowExtendedSTXSyntax or AllowCStrings
or AllowEStrings is true
For now: only use c-style, since stc does not support others.
'c' - C-style escapes:
\n,\t,\r,\b,\xXX,\uXXXX,\UXXXXXX,

much like character escapes in C-literals;
expands:
\n newLine
\r return
\t tab
\b backspace
\f formfeed
\g bell
\e escape
\0 nul

\\ backSlash
\ ...\ (backslash-separator) ignored up to next backslash
\xNN hexCharacter
\uNNNN hex UnicodeCharacter
\UNNNNNN hex UnicodeCharacter
\<name> named character
ignoreErrors: return the flag which controls notification of errors
ignoreErrors: aBoolean: enable/disable notification of errors
ignoreWarnings: return the flag which controls notification of warnings
ignoreWarnings: aBoolean: enable/disable notification of warnings
isCommentCharacter: ch: return true, if ch is the comment-start character.
Brought into a separate method to allow for easier subclassing
isDoIt
isSpecialOrExtendedSpecialCharacter: ch: self basicNew isSpecialOrExtendedSpecialCharacter:$-
notifying: anObject: set the requestor to be notified
setNameSpace: aNameSpace: ignored here
setPackage: aNameSpace: ignored here
setSyntax: aSyntax

reading next token

continueEscapedString: after escaping out of a string with \{,
an expression was read by the parser and it found the terminating $}.
It calls this to continue reading the string...
nextAssignmentArrow: return a left-arrow
nextBacktickIdentifier: a single back-quote has been scanned;
scan up to the next back-tick, and return it as an identifier
nextCharacter: a $ has been read - return a character token
nextColonOrAssign: colon has been read - look for = to make it an assign
nextExcla: a ! has been read - return either
the ! binarySelector,
ExclaLeftParen (for '!('),
ExclaLeftBrack (for '!['),
ExclaLeftBrace (for '!{')
nextExtendedSpecial: ch
nextExtendedString: delimiter escapeStyle: escapeStyleOrNil: a quote has been scanned; scan the string (caring for doubled quotes).
escapeStyle may be:
nil - old style ST80 strings (no character escapes)
or a single char string:
'c' - C-style escapes:
\n,\t,\r,\b,\xXX,\uXXXX,\UXXXXXX,

'e' - C-style with embedded expressions:
e'...{ expr1 } ... { exprN }' will generate:
'...%1 ... %N' bindWithArguments:{ expr1 . ... . exprN }

'i' - C-style plus embedded expressions internationalized:
i'...{ expr1 } ... { exprN }' will generate:
resources string:'...%1 ... %N' withArguments:{ expr1 . ... . exprN }

'r' - regex

'x' - extended-style escapes:
as yet unsupported
nextExtendedUnary: ch
nextHash: a # has been read - return either
a symbol,
HashLeftParen (for '#('),
HashLeftBrack (for '#['),
HashLeftBrace (for '#{' and AllowQualifiedNames/inline objects)
HashHashLeftParen (for '##(' and AllowDolphinExtensions)
HashHashLeftBrack (for '##[' )
HashHash (for '##' )

extended syntax (scheme-style literal arrays):
(requires ParserFlags allowSTXExtendedArrayLiterals:true)
HashTypedArrayParen (for '#u8(', '#s8(' , '#u16(' ...)
type in tokenValue: u1, u8, u16, u32, u64, s8, s16, s32, s64,
f16, f32, f64, fb16, f, d, b, B
nextId: no longer used here - remains for backwardCompatibility for
subclass users ... (sigh)
nextIdentifier: an alpha character
(or underscore if AllowUnderscore,
or greek if allowGreekCharactersInIdentifier
or national letter if allowNationalCharactersInIdentifier)
has been read.
Return the next identifier.
nextMantissa: radix: read the mantissa of a radix number

** This is an obsolete interface - do not use it (it may vanish in future versions) **
nextMantissaAndScaledPartWithRadix: radix: read the mantissa of a radix number.
Since we don't know yet if this is for a Float, LongFloat or a FixedPoint,
return info as a tuple which is useful for all:
- the mantissa as float/longFloat,
- the numerator
- and the scale for a fixedPoint.
If it turns out, that the number of digits is too many for Float precision,
the caller may still compute it using the numerator and the scale
(i.e. numerator asHighPrecisionFloat / (10 raisedTo:scale)
nextNumber: scan a number; handles radix prefix, mantissa and exponent.
Allows for
e, d or q to be used as exponent limiter (for float or long float),
s for scaled fixpoint numbers,
f for single precision floats (controlled by parserFlags),
Q for quad float,
QO for octa float,
QD for qDouble float,
QL for LargeFloats.

i.e. 1e5 -> float (technically a double precision IEEE)
1d5 -> float (also, a double precision IEEE)
1q5 -> long float (a c-long double / extended or quad precision IEEE, dep. on CPU)
1Q5 -> quad float (quad precision IEEE)
1QD5 -> qDouble float (4*double precision)
1QL5 -> large float (arbitrary precision)
1s -> a fixed point with precision from number of digits given.
1s5 -> a fixed point with 5 digits precision.
1d -> float (technically a double precision IEEE float).
1q -> long float (technically a c-long double / extended or quad precision IEEE float, dep. on CPU).
1Q -> quad float (quad precision IEEE)

1f5 -> shortFloat (technically a single precision IEEE float) or float, dep on parserFlags.
1f -> shortFloat (technically a single precision IEEE float) or float, dep on parserFlags.
support for scaled decimals can be disabled, if code needs to be read,
which does not know about them (very unlikely).
support for single prec. floats with f/F is controlled by a parser flag
nextPrimitive: scan an inline C-primitive.
nextSpecial: a special character has been read, look for another one.
also -number is handled here
nextSpecialWith: firstChar: a special character has been read, look for another one.
also -number is handled here
nextString: delimiter: a quote has been scanned; scan the string (caring for doubled quotes)
nextString: delimiter escapeStyle: escapeStyle: a quote has been scanned; scan the string (caring for doubled quotes).
escapeStyle may be:
nil - old style ST80 strings (no character escapes)
or a single char string:
'c' - C-style escapes:
\n,\t,\r,\b,\xXX,\uXXXX,\UXXXXXX,

'e' - C-style plus embedded escapes:
e'...{ expr1 } ... { exprN }' will generate:
'...%1 ... %N' bindWithArguments:{ expr1 . ... . exprN }

'i' - internationalized-string escapes:

'x' - extended-style escapes:
as yet unsupported
'r' - regex
as yet unsupported
nextSymbolAfterHash: helper: a # has been read - return #Symbol token or nil
nextSymbolAfterHash: prefix: helper: #<prefix> has been read - return #Symbol token or nil
nextToken: return the next token from the source-stream
nextToken: aCharacter: return a character token
nextUnderline: return either an identifier, or an underline (sometimes an assignment) token
skipComment: skip over a comment;
handles ST/X eol comments (dquote-slash)
and multiline-delimiter comment (dquote-less-less).
Multiline-delimiter comments start with dquote<<TOKEN and end in a line which starts with TOKEN, where TOKEN is any word with no separators
skipToEndOfLineRememberingIn: commentStreamOrNil
xnextString: delimiter escapeStyle: escapeStyleOrNil: marked as obsolete by exept MBP at 18-04-2023

** This is an obsolete interface - do not use it (it may vanish in future versions) **

Private classes:

    Comment
    Directive

ST/X 7.7.0.0; WebServer 1.702 at 20f6060372b9.unknown:8081; Mon, 18 Nov 2024 04:15:35 GMT

Smalltalk/X Webserver

Documentation of class 'Scanner':

Class: Scanner

Inheritance:

Description:

copyright

bugs

Class protocol:

Instance protocol:

Private classes: