|
Class: JavaScriptScanner
Object
|
+--Scanner
|
+--JavaScriptScanner
|
+--JavaScriptParser
- Package:
- stx:libjavascript
- Category:
- Languages-JavaScript-Compiling & Parsing
- Version:
- rev:
1.71
date: 2019/07/31 15:28:10
- user: cg
- file: JavaScriptScanner.st directory: libjavascript
- module: stx stc-classLibrary: libjavascript
tokenizer for JavaScript
news:
does regex '/pattern/' vs. division operator
determination via a rule described in
https://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
if that makes problems, set the variable:
SupportRegex
to false, to get the previous behavior (which is: not supporting regex)
class initialization
-
initialize
-
super initialize. -- no need for that
initialization
-
setupActions
-
initialize the scanners actionTables - these are used to dispatch
into scanner methods as characters are read
Notice, this is only called if ActionArray is nil.
ActionArray := nil
usage example(s):
JavaScriptScanner setupActions
|
accessing
-
tokenType
-
initialization
-
initialize
-
initialize the scanner
private
-
checkForKeyword: string
-
check if string is a keyword (as opposed to an identifier).
-
handleCategoryDirective: categoryString
-
called when encountering a /** category: xxxx **/ comment;
categoryString will be xxxx.
Can be redefined in subclasses
-
handleCommentDirectivesIn: commentText
-
called for the text after the initial "/*"
-
isCommentCharacter: ch
-
return true, if ch is the comment-start character
-
previousTokenInitiatesRegexForSlash
-
read
https://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
for an explanation
reading next token
-
hex2CharacterEscape
-
-
hex4CharacterEscape
-
-
nextCharacter
-
this is a special (non-JavaScript) extension to support $'x' for character literals
-
nextMulti: list after: firstChar
-
a char has been read - peek ahead in list
-
nextNumber
-
nextChar == (Character cr) ifTrue:[
-
nextRegex
-
notice: initial '/' has already been read
-
nextSlash
-
-
nextString
-
-
nextStringOrCharacter: asCharacter
-
-
nextStringOrCharacter: asCharacter orRegex: asRegex
-
skip over initial quote
-
nextSymbol
-
this is a special (non-JavaScript) extension to support #'x' for symbol literals
-
nextToken
-
Verbose := true
-
skipComment
-
* has already been read"
-
skipEOLComment
-
/ has already been read"
<<END
|s in|
in := '
function scalefactor(value) {
scalevector[0]=value;
scalevector[1]=1.;
scalevector[2]=1.;
}
'.
s := JavaScriptScanner for:in readStream.
s nextToken
|
|s in|
in := '
function scalefactor(value) {
scalevector[0]=value;
scalevector[1]=1.;
scalevector[2]=1.;
}
'.
s := JavaScriptScanner new.
s scanTokens:(in readStream).
|
|s in|
in := '
function scalefactor(value) {
return "hello";
}
'.
s := JavaScriptScanner new.
s scanTokens:(in readStream).
|
|s in|
in := '
function scalefactor(value) {
return /hello/;
}
'.
in := in readStream.
s := JavaScriptScanner for:in.
[in atEnd] whileFalse:[
Transcript showCR:s nextToken
]
|
|s in|
in := '
function scalefactor(value) {
return a /hello/;
}
'.
in := in readStream.
s := JavaScriptScanner for:in.
[in atEnd] whileFalse:[
Transcript showCR:s nextToken
]
| END"
|