eXept Software AG Logo

Smalltalk/X Webserver

Documentation of class 'Scanner':

Home

Documentation
www.exept.de
Everywhere
for:
[back]

Class: Scanner


Inheritance:

   Object
   |
   +--Scanner
      |
      +--JavaScriptScanner
      |
      +--Parser
      |
      +--SunRPC::XDRScanner

Package:
stx:libcomp
Category:
System-Compiler
Version:
rev: 1.399 date: 2019/08/17 12:36:53
user: cg
file: Scanner.st directory: libcomp
module: stx stc-classLibrary: libcomp
Author:
Claus Gittinger

Description:


Scanner reads from a stream and returns individual smalltalk tokens
Its main method is #nextToken, which reads and returns the next token
Possibly placing additional information (such as tokenValue) into
instance variables as a side effect.

TODO:
    some testers noticed that ST-80's scanner methods are called
    xLetter, xDigit etc. For code using these (internals), the nextNumber,
    nextIdentifier etc. methods should be renamed.
    (to me, these seem to be internal private methods;
     their public use is not a good idea ..)

    Scanner is typically subclassed for parsing and #nextToken
    invoked via #self-sends.
    This should be changed and scanner ought to be an instance variable
    of Parser - this allows more flexible use of the scanner/parser
    framework (i.e. changing the scanner without affecting the parser).

Extensions:
    this scanner allows for 3-character binary selectors.
    also, # is a valid selector. (however, ## is currently scanned as a symbol literal).


Related information:

    Parser

Class protocol:

Signal constants
o  doNotShowCompilerWarningAgainActionQuery

o  emptySourceNotificationSignal

accessing
o  actionArray

o  flushActionArray
Scanner flushActionArray
Parser flushActionArray
ByteCodeCompiler flushActionArray
Explainer flushActionArray

o  typeArray

o  unicodeActions

class initialization
o  initialize
initialize the classes defaults. Typically, these are changed
later in the 'private.rc' file.

usage example(s):

     self initialize

defaults
o  allowDollarInIdentifier
return true, if $-characters are allowed in identifiers.
Notice, that dollars are NEVER allowed as the first character in an identifier.

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  allowDollarInIdentifier: aBoolean
this allows turning on/off $-characters in identifiers.
Notice, that dollars are NEVER allowed as the first character in an identifier.
If turned off (the default), dollars are not allowed in identifiers,
but instead are scanned as character-constant prefix.
If turned on, dollars are in identifiers are allowed, while extra
dollars are still scanned as constant character prefix.
If you have to fileIn old VW-Vsn2.x classes, turn this off
before filing them in; i.e.:
Compiler allowDollarInIdentifiers:false

o  allowDolphinExtensions
return true, if ##(..) computed literal Arrays are allowed

o  allowDolphinExtensions: aBoolean
this allows turning on/off support for computed literal Arrays ##(..) as in dolphin.
If you want to fileIn Dolphin classes, enable this with:
Compiler allowDolphinComputedArrays:true

usage example(s):

     self allowDolphinExtensions:true
     self allowDolphinExtensions:false

o  allowLiteralNameSpaceSymbols
return true, if literal nameSpace symbols are allowed (#foo::bar) are allowed

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  allowOldStyleAssignment
return true, if underscore-assignment (pre ST-80v4 syntax) are to be allowed

o  allowOldStyleAssignment: aBoolean
this allows turning on/off recognition of underscore-assignment (pre ST-80v4 syntax).
You must turn this off, if code with variables named '_' is to be filedIn

o  allowQualifiedNames
return true, if '#{..}' and 'namespace.varName' qualified names are allowed

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  allowQualifiedNames: aBoolean
this allows turning on/off support for qualifiedNames #{ .., } as in vw3.
If you want to fileIn vw3 or later classes, enable this with:
Compiler allowQualifiedNames:true
Notice, that qualified names are not really supported semantically
(they are parsed, but treated like regular globals)

usage example(s):

     self allowQualifiedNames:true
     self allowQualifiedNames:false

o  allowSqueakExtensions
return true, if support for squeak extensions
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)
is enabled.

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  allowSqueakExtensions: aBoolean
this allows turning on/off support for squeak extensions:
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)

If you want to fileIn Squeak classes, enable this with:
Compiler allowSqueakComputedArrays:true

usage example(s):

     self allowSqueakExtensions:true
     self allowSqueakExtensions:false

o  allowUnderscoreInIdentifier
return true, if underscores are allowed in identifiers

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  allowUnderscoreInIdentifier: aBoolean
this allows turning on/off underscores in identifiers.
If turned off (the default), underscores are not allowed in identifiers,
but instead scanned as assignment character (old ST/80 syntax).
If turned on, underscores are in identifiers are allowed, while extra
underscores are still scanned as assignment.
If you have to fileIn old VW-Vsn2.x classes,
turn them off with:
Compiler allowUnderscoreInIdentifiers:false

o  maxBinarySelectorSize

o  warnCommonMistakes
return true, if common beginners mistakes are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  warnCommonMistakes: aBoolean
this allows turning on/off warnings about common beginners mistakes.
Those are not really errors in the strict sense, but often lead to
run time errors later.
Examples are: expr or:expr2, where expr2 is not a block.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnCommonMistakes:false
in your 'private.rc' file

o  warnDollarInIdentifier
return true, if $-characters in identifiers are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  warnDollarInIdentifier: aBoolean
this allows turning on/off warnings about $-characters in identifiers.
You may find those warnings useful, to make certain that your code
is portable to other smalltalk versions, which do not allow this
(i.e. VW releases 2.x and maybe others).
Notice, that dollars are NEVER allowed as the first character in an identifier.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnDollarInIdentifier:false
in your 'private.rc' file

o  warnOldStyleAssignment
return true, if underscore-assignment (pre ST-80v4 syntax) are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  warnOldStyleAssignment: aBoolean
this allows turning on/off warnings about underscore-assignment (pre ST-80v4 syntax).
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnOldStyleAssignment:false
in your 'private.rc' file

o  warnPossibleIncompatibilities
return true, if possible incompatibilities (with other ST systems)
are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  warnPossibleIncompatibilities: aBoolean
this turns warnings about possible incompatibilities (with other ST systems)
on or off.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnPossibleIncompatibilities:false
in your 'private.rc' file.

o  warnSTXSpecials
return true, if ST/X specials are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  warnSTXSpecials: aBoolean
this allows turning on/off warnings about stx specials.
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnSTXSpecials:false
in your 'private.rc' file

o  warnUnderscoreInIdentifier
return true, if underscores in identifiers are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  warnUnderscoreInIdentifier: aBoolean
this allows turning on/off warnings about underscores in identifiers.
You may find those warnings useful, to make certain that your code
is portable to other smalltalk versions, which do not allow this
(i.e. VW releases 2.x).
If you get bored by those warnings, turn them off by adding
a line as:
Compiler warnUnderscoreInIdentifier:false
in your 'private.rc' file

o  warnings
return true, if any warnings are to be shown

o  warnings: aBoolean
this allows turning on/off all warnings; the default is on.
You can turn off warnings in your 'private.rc' file with
Compiler warnings:false

initialization
o  binarySelectorCharacters
return a collection of characters which are allowed in binary selectors

o  extendedBinarySelectorCharacters
return a collection of characters which are optionally allowed in binary selectors

o  setupActions
initialize the scanner's actionTables - these are used to dispatch
into scanner methods as characters are read.
Compatibility note: in previous versions, these tables used to be kept
in classVariables, which made reuse hard as subclasses had no easy way of
defining their own tables. These are now class instance variables.

usage example(s):

     Scanner setupActions
     Scanner withAllSubclassesDo:[:cls | cls setupActions ]

instance creation
o  for: aStringOrStream
create & return a new scanner reading from aStringOrStream

o  new
create & return a new scanner

utility scanning
o  scanNumberFrom: aStream
utility - helper for Number>>readSmalltalkSyntaxFrom:


Instance protocol:

Compatibility-ST80
o  endOfLastToken
return the position of the token which was just read.
This method was required by some PD program.
It is not maintained and may be removed without notice.

o  scan: aStringOrStream
initialize the scanner: set the source-stream and
preread the first token

o  scanDoing: aBlock
scan, evaluating aBlock for every scanned token.

o  scanToken
read the next token from my input stream

o  scanTokens: aStringOrStream
return a collection of symbolic tokens from the passed input,
similar to what would be read when reading '#( <string> )'

usage example(s):

     Scanner new scanTokens:'name ident #sym ''string'' 8r12'

     Scanner new scanTokens:'translate (200px 100px)'

     Scanner new
	scanTokens:'Boolean subclass:#True
				instanceVariableNames:''''
				classVariableNames:''''
				poolDictionaries:''''
				category:''Kernel-Objects''
	'

Compatibility-Squeak
o  notify: message at: codeStreamPosition
Squeak compatibility

accessing
o  atEnd
true if the scanned source stream's end has been reached

o  comments

o  exitBlock: aBlock

o  expectToken: expectedToken

o  getCollectedComments
retrieve the so far collected comments, reset comment collection

o  hasError
return true if there where any errors (valid after parsing)

o  inArrayLiteral: aBoolean

o  lineNumber

o  newSourceStream: aStream

o  outStream: aStream
if set, this stream gets the parsed comments as it is read.
Useful to collect comments while reading

o  requestor

o  saveComments: aBoolean

o  sourcePosition

o  sourcePositionWithoutPeeks

o  sourceStream

o  tokenEndPosition

o  tokenLineNr

o  tokenName

o  tokenPosition

o  tokenStartPosition

o  tokenType

o  tokenValue

accessing-flags
o  allowDollarInIdentifier

o  allowDollarInIdentifier: something

o  allowLiteralNameSpaceSymbols

o  allowLiteralNameSpaceSymbols: aBoolean

o  allowOldStyleAssignment

o  allowOldStyleAssignment: aBoolean

o  allowSqueakExtensions
return true, if support for squeak extensions
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)
is enabled.

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  allowSqueakExtensions: aBoolean
this allows turning on/off support for squeak extensions:
computed arrays { .., }
c/java style arguments in message sends rec foo(arg1, ... argN)

o  allowUnderscoreInIdentifier

o  allowUnderscoreInIdentifier: aBoolean

o  didWarnAboutOldStyleAssignment

o  parserFlags

o  parserFlags: aParserFlagsInstance

o  scanColonAsKeyword
not used here, but eases subclassing for other languages.

o  warnCommonMistakes

o  warnCommonMistakes: aBoolean

o  warnDollarInIdentifier

o  warnDollarInIdentifier: aBoolean

o  warnOldStyleAssignment

o  warnOldStyleAssignment: aBoolean

o  warnPossibleIncompatibilities
return true, if possible incompatibilities (with other ST systems)
are to be warned about

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  warnPossibleIncompatibilities: aBoolean

o  warnSTXNameSpaceUse

o  warnSTXNameSpaceUse: aBoolean

o  warnSTXSpecialComment

o  warnSTXSpecialComment: aBoolean

o  warnUnderscoreInIdentifier

o  warnUnderscoreInIdentifier: aBoolean

directives
o  parseClassDirective
Class: className

o  parseClassHintDirective
ClassHint: className

o  parseDirective
parse a directive inside a comment (introduced with '{').
This is an ST/X special

o  parseDirectiveClassNameArg
helper for parsing a directive

o  parseDirectiveStringArg
helper for parsing a directive.
scans a word starting with a letter or underline,
and consisting of letters, digits, underlines or dots.

o  parseDirectiveStringArg: firstCharacterCheckBlock rest: restCharacterCheckBlock
helper for parsing a directive

o  parseDirectiveStringListArg
helper for parsing a directive

o  parseNamespaceDirective
Namespace: 'nameSpace'
Namespace: nameSpace

o  parsePackageDirective
Package: 'name-of-package'
Package: packageId

o  parsePrerequisitesDirective
Prerequisites: 'name-of-package1', ... , 'name-of-packageN'

o  parseRequiresDirective
Require: 'name-of-feature', ... , 'name-of-featureN'

o  parseSyntaxDirective
Syntax: 'st-syntax-id'

o  parseUsesDirective
Uses: 'nameSpace1', ... , 'nameSpaceN'
Uses: nameSpaceId1, ... , nameSpaceIdN

dummy-syntax highlighting
o  markCommentFrom: pos1 to: pos2

o  markConstantFrom: pos1 to: pos2

o  markErrorFrom: pos1 to: pos2

o  markStringFrom: pos1 to: pos2

o  markSymbolFrom: pos1 to: pos2

error handling
o  clearErrorFlag

o  correctableError: message position: pos1 to: pos2
report an error which can be corrected by compiler -
return non-false, if correction is wanted
(there is more than true/false returned here)

o  correctableSelectorWarning: message position: pos1 to: pos2
report a warning which can be corrected by compiler -
return non-false, if correction is wanted (there is more than
true/false returned here)

o  correctableWarning: aMessage doNotShowAgainAction: doNotShowAgainAction doNotShowAgainForThisMethodAction: doNotShowAgainForThisMethodAction position: position to: endPos
notify requestor of a warning - if there is no requestor, just ignore it.
Return the result passed back from the requestor (or false, if there is none).

o  correctableWarning: message position: pos1 to: pos2
report an error which can be corrected by compiler -
return non-false, if correction is wanted
(there is more than true/false returned here - i.e. a symbol, describing how to fix it)

o  disableWarningsOnCurrentMethodFor: flagName
ignored here

o  errorFlag: flagArg

o  errorMessagePrefix

o  ignorableParseError: message
if proceeded, install method anyway (used with non-st/x primitives)

o  invalidCharacter: ch

o  lastTokenLineNumber
return the line number of the token which was just read.

o  notifyError: aMessage position: position to: endPos
notify requestor of an error - if there is no requestor
put it on the transcript. Requestor is typically the CodeView
in which the accept/doIt was triggered, or the PositionableStream
which does the fileIn. The requestor may decide how to highlight the
error (and/or to abort the compile).
Return the result passed back by the requestor.

o  notifyError: aMessage position: position to: endPos lineNr: lineNrOrNil
notify requestor of an error - if there is no requestor
put it on the transcript. Requestor is typically the CodeView
in which the accept/doIt was triggered, or the PositionableStream
which does the fileIn. The requestor may decide how to highlight the
error (and/or to abort the compile).
Return the result passed back by the requestor.

o  notifyWarning: aMessage doNotShowAgainAction: doNotShowAgainAction doNotShowAgainForThisMethodAction: doNotShowAgainForThisMethodAction position: position to: endPos
notify requestor of a warning - if there is no requestor, just ignore it.
Return the result passed back from the requestor (or false, if there is none).

o  notifyWarning: aMessage doNotShowAgainAction: doNotShowAgainAction position: position to: endPos
notify requestor of a warning - if there is no requestor, just ignore it.
Return the result passed back from the requestor (or false, if there is none).

o  notifyWarning: aMessage position: position to: endPos
notify requestor of an warning - if there is no requestor
put it on the transcript.
Return the result passed back by the requestor.

o  parseError: aMessage
report an error

o  parseError: aMessage line: lNr
report an error

o  parseError: aMessage position: position
report an error

o  parseError: aMessage position: position to: endPos
report an error

o  positionFromLineNumber: lNr

o  setErrorFlag

o  showErrorMessage: aMessage position: pos
show an errormessage on the Transcript

o  syntaxError: aMessage
a syntax error happened - position is not known

o  syntaxError: aMessage position: position
a syntax error happened - only start position is known

o  syntaxError: aMessage position: position to: endPos
a syntax error happened.
Return true for correction, false if not

o  warnCommonMistake: msg at: position
warn about a common beginners mistake

o  warnCommonMistake: msg position: pos1 to: pos2
warn about a common beginners mistake

o  warnDollarAt: position
warn about $-character in an identifier

o  warnOldStyleAssignmentAt: position
warn about an oldStyle assignment

o  warnParagraphAt: position
warn about §-character in an identifier

o  warnPeriodAt: position
warn about a period in an identifier

o  warnPossibleIncompatibility: msg position: pos1 to: pos2
warn about a possible incompatibility with other ST systems

o  warnSTXSpecialCommentAt: position to: endPosition
dfo

o  warnUnderscoreAt: position
warn about an underscore in an identifier

o  warning: aMessage
a warning - position is not known

o  warning: aMessage doNotShowAgainAction: doNotShowAgainAction doNotShowAgainForThisMethodAction: doNotShowAgainForThisMethodAction position: position to: endPos
a warning

o  warning: aMessage doNotShowAgainAction: doNotShowAgainAction position: position to: endPos
a warning

o  warning: aMessage line: lNr
a warning - only start position is known

o  warning: aMessage position: position
a warning - only start position is known

o  warning: aMessage position: position to: endPos
a warning

o  warningMessagePrefix

general scanning
o  scanNumberFrom: aStringOrStream
scan aSourceString for the next number in smalltalk syntax.
Return the number or nil. Leave the position of the stream after the number
or unchanged (if no number is coming)
Extension:
since constant fractions (int/int) are treated like lexical tokens (not messages)
in Smalltalk/X, this also reads fractions.

o  scanPositionsFor: aTokenString inString: aSourceString
scan aSourceString for occurrences of aTokenString.
Return a collection of start positions.
Added for VW compatibility (to support simple syntax-highlight).

usage example(s):

     Scanner new scanPositionsFor:'hello' inString:'foo bar hello baz hello'
     Scanner new scanPositionsFor:'3.14' inString:'foo 3.145 bar hello 3.14 baz hello 3.14'
     Scanner new scanPositionsFor:'16' inString:'foo 16 bar hello 16r10 baz hello 2r10000'

initialization
o  initialize
initialize the scanner

o  initializeActionTable

o  initializeFlagsFrom: aScanner
initialize flags from another scanner

o  initializeFor: aStringOrStream
initialize the new scanner & prepare for reading from aStringOrStream

o  requestor: anObject
set the requestor to be notified

o  setSource: newSourceStream

o  source: aStringOrStream
prepare for reading from aStringOrStream;
notice: if token is nonNil, it is preserved. This allows for scanning
across streams.

parser interface
o  token

o  tokenLineNr: lineNumberArg

o  tokenPosition: positionArg

private
o  backupPosition
if reading from a stream, at the end we might have read
one token too many

o  beginComment

o  characterNamed: name

o  checkForKeyword: string
check if string is a keyword (as opposed to an identifier).
That is, its one of 'self', 'super', 'nil', 'true', 'false',
or 'thisContext'.
'here' is handled elsewhere (since it must be treated as an
identifier, if declared locally.

o  collectedSource

o  eatPeekChar

o  endComment: comment

o  endComment: commentString type: commentType
obsolete; left for backward compatibility

o  endComment: commentString type: commentType start: startPos end: endPos

o  eolIsWhiteSpace

o  escapeCharacterFor: aCharacter escapeStyle: escapeStyle
only if AllowExtendedSTXSyntax or AllowCStrings
or AllowEStrings is true
For now: only use c-style, since stc does not support others.
'c' - C-style escapes:
\n,\t,\r,\b,\xXX,\uXXXX,\UXXXXXX,

much like character escapes in C-literals;
expands:
\n newLine
\r return
\t tab
\b backspace
\f formfeed
\g bell
\0 nul

\\ backSlash
\ ...\ (backslash-separator) ignored up to next backslash
\xNN hexCharacter
\uNNNN hex UnicodeCharacter
\UNNNNNN hex UnicodeCharacter
\<name> named character

o  ignoreErrors
return the flag which controls notification of errors

o  ignoreErrors: aBoolean
enable/disable notification of errors

o  ignoreWarnings
return the flag which controls notification of warnings

o  ignoreWarnings: aBoolean
enable/disable notification of warnings

o  isCommentCharacter: ch
return true, if ch is the comment-start character.
Brought into a separate method to allow for easier subclassing

o  isDoIt

o  isSpecialOrExtendedSpecialCharacter: ch
self basicNew isSpecialOrExtendedSpecialCharacter:$-

o  notifying: anObject
set the requestor to be notified

o  setNameSpace: aNameSpace
ignored here

o  setPackage: aNameSpace
ignored here

o  setSyntax: aSyntax

reading next token
o  continueEscapedString
after escaping out of a string with \{,
an expression was read by the parser and it found the terminating $}.
It calls this to continue reading the string...

o  nextAssignmentArrow
return a left-arrow

o  nextBacktickIdentifier
a single back-quote has been scanned;
scan up to the next back-tick, and return it as an identifier

o  nextCharacter
a $ has been read - return a character token

o  nextColonOrAssign
colon has been read - look for = to make it an assign

o  nextExcla
a ! has been read - return either
the ! binarySelector,
ExclaLeftParen (for '!('),
ExclaLeftBrack (for '!['),
ExclaLeftBrace (for '!{')

o  nextExtendedSpecial: ch

o  nextHash
a # has been read - return either
a symbol,
HashLeftParen (for '#('),
HashLeftBrack (for '#['),
HashLeftBrace (for '#{' and AllowQualifiedNames)
HashHashLeftParen (for '##(' and AllowDolphinExtensions)
HashHashLeftBrack (for '##[' )
HashHash (for '##' )

extended syntax (scheme-style literal arrays):
(requires ParserFlags allowSTXExtendedArrayLiterals:true)
HashTypedArrayParen (for '#u8(', '#s8(' , '#u16(' ...)
type in tokenValue: u1, u8, u16, u32, u64, s8, s16, s32, s64,
f16, f32, f64, f, d, b, B

o  nextId
no longer used here - remains for backwardCompatibility for
subclass users ... (sigh)

o  nextIdentifier
an alpha character
(or underscore if AllowUnderscore,
or greek if allowGreekCharactersInIdentifier
or national letter if allowNationalCharactersInIdentifier)
has been read.
Return the next identifier.

o  nextMantissa: radix
read the mantissa of a radix number

** This is an obsolete interface - do not use it (it may vanish in future versions) **

o  nextMantissaAndScaledPartWithRadix: radix
read the mantissa of a radix number.
Since we don't know yet if this is for a Float, LongFloat or a FixedPoint,
return info which is useful for all: an array consisting of
the mantissa as float/longFloat, the numerator and the scale for a fixedPoint.

o  nextNumber
scan a number; handles radix prefix, mantissa and exponent.
Allows for
e, d or q to be used as exponent limiter (for float or long float),
s for scaled fixpoint numbers,
f for single precision floats (controlled by parserFlags).

i.e. 1e5 -> float (technically a double precision IEEE)
1d5 -> float (also, a double precision IEEE)
1q5 -> long float (a c-long double / quad precision IEEE)
1s -> a fixed point with precision from number of digits given.
1s5 -> a fixed point with 5 digits precision.
1d -> float (technically a double precision IEEE float).
1q -> long float (technically a c-long double / extended or quad precision IEEE float).

1f5 -> shortFloat (technically a single precision IEEE float) or float.
1f -> shortFloat (technically a single precision IEEE float) or float.
support for scaled decimals can be disabled, if code needs to be read,
which does not know about them (very unlikely)

o  nextPrimitive
scan an inline C-primitive.

o  nextSpecial
a special character has been read, look for another one.
also -number is handled here

o  nextSpecialWith: firstChar
a special character has been read, look for another one.
also -number is handled here

o  nextString: delimiter
a quote has been scanned; scan the string (caring for doubled quotes)

o  nextString: delimiter escapeStyle: escapeStyle
a quote has been scanned; scan the string (caring for doubled quotes).
escapeStyle may be:
nil - old style ST80 strings (no character escapes)
ot a single char string:
'c' - C-style escapes:
\n,\t,\r,\b,\xXX,\uXXXX,\UXXXXXX,

'e' - C-style plus embedded escapes:
e'...{ expr1 } ... { exprN }' will generate:
'...%1 ... %N' bindWithArguments:{ expr1 . ... . exprN }

'x' - extended-style escapes:
as yet unsupported
'r' - regex
as yet unsupported


o  nextSymbolAfterHash
helper: a # has been read - return #Symbol token or nil

o  nextSymbolAfterHash: prefix
helper: #<prefix> has been read - return #Symbol token or nil

o  nextToken
return the next token from the source-stream

o  nextToken: aCharacter
return a character token

o  nextUnderline
return either an identifier, or an underline (sometimes an assignment) token

o  skipComment
skip over a comment;
handles ST/X eol comments (quote-slash)
and multiline-delimiter comment (quote-less-less).
Multiline-delimiter comments start with ''<<TOKEN and end in a line which starts with TOKEN (quote-quote being a double quote here)

o  skipToEndOfLineRememberingIn: commentStreamOrNil

o  xnextString: delimiter escapeStyle: escapeStyle
a quote has been scanned; scan the string (caring for doubled quotes).
escapeStyle may be:
nil - old style ST80 strings (no character escapes)
or a single char string:
'c' - C-style escapes:
\n,\t,\r,\b,\xXX,\uXXXX,\UXXXXXX,

'e' - C-style plus embedded escapes:
e'...{ expr1 } ... { exprN }' will generate:
'...%1 ... %N' bindWithArguments:{ expr1 . ... . exprN }

'r' - regex

'x' - extended-style escapes:
as yet unsupported


Private classes:

    Comment
    Directive


ST/X 7.2.0.0; WebServer 1.670 at bd0aa1f87cdd.unknown:8081; Mon, 04 Jul 2022 12:40:01 GMT