eXept Software AG Logo

Smalltalk/X Webserver

Documentation of class 'JavaScriptScanner':

Home

Documentation
www.exept.de
Everywhere
for:
[back]

Class: JavaScriptScanner


Inheritance:

   Object
   |
   +--Scanner
      |
      +--JavaScriptScanner
         |
         +--JavaScriptParser

Package:
stx:libjavascript
Category:
Languages-JavaScript-Compiling & Parsing
Version:
rev: 1.71 date: 2019/07/31 15:28:10
user: cg
file: JavaScriptScanner.st directory: libjavascript
module: stx stc-classLibrary: libjavascript

Description:


tokenizer for JavaScript

news:
    does regex '/pattern/' vs. division operator 
    determination via a rule described in
        https://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
    if that makes problems, set the variable:
        SupportRegex
    to false, to get the previous behavior (which is: not supporting regex)


Class protocol:

class initialization
o  initialize
super initialize. -- no need for that

initialization
o  setupActions
initialize the scanners actionTables - these are used to dispatch
into scanner methods as characters are read
Notice, this is only called if ActionArray is nil.
ActionArray := nil

usage example(s):

     JavaScriptScanner setupActions


Instance protocol:

accessing
o  tokenType

initialization
o  initialize
initialize the scanner

private
o  checkForKeyword: string
check if string is a keyword (as opposed to an identifier).

o  handleCategoryDirective: categoryString
called when encountering a /** category: xxxx **/ comment;
categoryString will be xxxx.
Can be redefined in subclasses

o  handleCommentDirectivesIn: commentText
called for the text after the initial "/*"

o  isCommentCharacter: ch
return true, if ch is the comment-start character

o  previousTokenInitiatesRegexForSlash
read
https://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
for an explanation

reading next token
o  hex2CharacterEscape

o  hex4CharacterEscape

o  nextCharacter
this is a special (non-JavaScript) extension to support $'x' for character literals

o  nextMulti: list after: firstChar
a char has been read - peek ahead in list

o  nextNumber
nextChar == (Character cr) ifTrue:[

o  nextRegex
notice: initial '/' has already been read

o  nextSlash

o  nextString

o  nextStringOrCharacter: asCharacter

o  nextStringOrCharacter: asCharacter orRegex: asRegex
skip over initial quote

o  nextSymbol
this is a special (non-JavaScript) extension to support #'x' for symbol literals

o  nextToken
Verbose := true

o  skipComment
* has already been read"

o  skipEOLComment
/ has already been read"


Examples:


<<END
    |s in|

    in := '
function scalefactor(value) {
    scalevector[0]=value;
    scalevector[1]=1.;
    scalevector[2]=1.;
}
'.

    s := JavaScriptScanner for:in readStream.
    s nextToken  
    |s in|

    in := '
function scalefactor(value) {
    scalevector[0]=value;
    scalevector[1]=1.;
    scalevector[2]=1.;
}
'.
    s := JavaScriptScanner new.
    s scanTokens:(in readStream).
    |s in|

    in := '
function scalefactor(value) {
    return "hello";
}
'.
    s := JavaScriptScanner new.
    s scanTokens:(in readStream).
    |s in|

    in := '
function scalefactor(value) {
    return /hello/;
}
'.
    in := in readStream.
    s := JavaScriptScanner for:in.
    [in atEnd] whileFalse:[
        Transcript showCR:s nextToken
    ]
    |s in|

    in := '
function scalefactor(value) {
    return a /hello/;
}
'.
    in := in readStream.
    s := JavaScriptScanner for:in.
    [in atEnd] whileFalse:[
        Transcript showCR:s nextToken
    ]
END"

ST/X 7.2.0.0; WebServer 1.670 at bd0aa1f87cdd.unknown:8081; Tue, 17 May 2022 22:01:14 GMT