Class: JavaScriptScanner

• Inheritance
• Description
• Class protocol
  • class initialization
  • initialization
• Instance protocol
  • accessing
  • initialization
  • private
  • reading next token
• Examples

Inheritance:

   Object
   |
   +--Scanner
      |
      +--JavaScriptScanner
         |
         +--JavaScriptParser

Package:

stx:libjavascript

Category:

Languages-JavaScript-Compiling & Parsing

Version:

rev: 1.71 date: 2019/07/31 15:28:10

user: cg

file: JavaScriptScanner.st directory: libjavascript

module: stx stc-classLibrary: libjavascript

Description:

tokenizer for JavaScript

news:
    does regex '/pattern/' vs. division operator 
    determination via a rule described in
        https://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
    if that makes problems, set the variable:
        SupportRegex
    to false, to get the previous behavior (which is: not supporting regex)

Class protocol:

 initialize

super initialize. -- no need for that

 setupActions

initialize the scanners actionTables - these are used to dispatch
into scanner methods as characters are read
Notice, this is only called if ActionArray is nil.
ActionArray := nil

usage example(s):

JavaScriptScanner setupActions

Instance protocol:

 tokenType

 initialize

initialize the scanner

 checkForKeyword: string

check if string is a keyword (as opposed to an identifier).

 handleCategoryDirective: categoryString

called when encountering a /** category: xxxx **/ comment;
categoryString will be xxxx.
Can be redefined in subclasses

 handleCommentDirectivesIn: commentText

called for the text after the initial "/*"

 isCommentCharacter: ch

return true, if ch is the comment-start character

 previousTokenInitiatesRegexForSlash

read
https://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
for an explanation

 hex2CharacterEscape

 hex4CharacterEscape

 nextCharacter

this is a special (non-JavaScript) extension to support $'x' for character literals

 nextMulti: list after: firstChar

a char has been read - peek ahead in list

 nextNumber

nextChar == (Character cr) ifTrue:[

 nextRegex

notice: initial '/' has already been read

 nextSlash

 nextString

 nextStringOrCharacter: asCharacter

 nextStringOrCharacter: asCharacter orRegex: asRegex

skip over initial quote

 nextSymbol

this is a special (non-JavaScript) extension to support #'x' for symbol literals

 nextToken

Verbose := true

 skipComment

* has already been read"

 skipEOLComment

/ has already been read"

Examples:

    |s in|

    in := '
function scalefactor(value) {
    scalevector[0]=value;
    scalevector[1]=1.;
    scalevector[2]=1.;
}
'.

    s := JavaScriptScanner for:in readStream.
    s nextToken  

    |s in|

    in := '
function scalefactor(value) {
    scalevector[0]=value;
    scalevector[1]=1.;
    scalevector[2]=1.;
}
'.
    s := JavaScriptScanner new.
    s scanTokens:(in readStream).

    |s in|

    in := '
function scalefactor(value) {
    return "hello";
}
'.
    s := JavaScriptScanner new.
    s scanTokens:(in readStream).

    |s in|

    in := '
function scalefactor(value) {
    return /hello/;
}
'.
    in := in readStream.
    s := JavaScriptScanner for:in.
    [in atEnd] whileFalse:[
        Transcript showCR:s nextToken
    ]

    |s in|

    in := '
function scalefactor(value) {
    return a /hello/;
}
'.
    in := in readStream.
    s := JavaScriptScanner for:in.
    [in atEnd] whileFalse:[
        Transcript showCR:s nextToken
    ]