Class: JavaScriptScanner

• Inheritance
• Description
• Class protocol
  • class initialization
  • initialization
• Instance protocol
  • accessing
  • initialization
  • private
  • reading next token
• Examples

Inheritance:

   Object
   |
   +--Scanner
      |
      +--JavaScriptScanner
         |
         +--JavaScriptParser

Package:

stx:libjavascript

Category:

Languages-JavaScript-Compiling & Parsing

Version:

rev: 1.82 date: 2024/04/22 17:04:15

user: stefan

file: JavaScriptScanner.st directory: libjavascript

module: stx stc-classLibrary: libjavascript

Description:

tokenizer for JavaScript

news:
    does regex '/pattern/' vs. division operator 
    determination via a rule described in
        https://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
    if that makes problems, set the variable:
        SupportRegex
    to false, to get the previous behavior (which is: not supporting regex)

copyright
COPYRIGHT (c) 1998 by eXept Software AG
             All Rights Reserved

This software is furnished under a license and may be used
only in accordance with the terms of that license and with the
inclusion of the above copyright notice.   This software may not
be provided or otherwise made available to, or used by, any
other person.  No title to or ownership of the software is
hereby transferred.

Class protocol:

 initialize

super initialize. -- no need for that

 setupActions

initialize the scanners actionTables - these are used to dispatch
into scanner methods as characters are read
Notice, this is only called if ActionArray is nil.
ActionArray := nil

Usage example(s):

JavaScriptScanner setupActions JavaScriptParser setupActions JavaScriptCompiler setupActions JavaScriptSyntaxHighlighter setupActions self withAllSubclassesDo:#setupActions

Instance protocol:

 tokenType

 initialize

initialize the scanner

 checkForKeyword: string

check if string is a keyword (as opposed to an identifier).

 handleCategoryDirective: categoryString

called when encountering a /** category: xxxx **/ comment;
categoryString will be xxxx.
Can be redefined in subclasses

 handleCommentDirectivesIn: commentText

called for the text after the initial "/*"

 isCommentCharacter: ch

return true, if ch is the comment-start character

 previousTokenInitiatesRegexForSlash

read
https://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
for an explanation

 hex2CharacterEscape

 hex4CharacterEscape

 nextCharacter

this is a special (non-JavaScript) extension to support $'x' for character literals

 nextMulti: list after: firstChar

a char has been read - peek ahead in list

 nextNumber

nextChar == (Character cr) ifTrue:[

 nextRegex

notice: initial '/' has already been read

 nextSlash

 nextString

 nextStringOrCharacter: asCharacter

 nextStringOrCharacter: asCharacter orRegex: asRegex

skip over initial quote

 nextSymbol

this is a special (non-JavaScript) extension to support #'x' for symbol literals

 nextToken

Verbose := true

 skipComment

* has already been read"

 skipEOLComment

/ has already been read"

Examples:

    |s in|

    in := '
function scalefactor(value) {
    scalevector[0]=value;
    scalevector[1]=1.;
    scalevector[2]=1.;
}
'.

    s := JavaScriptScanner for:in readStream.
    s nextToken  

    |s in|

    in := '
function scalefactor(value) {
    scalevector[0]=value;
    scalevector[1]=1.;
    scalevector[2]=1.;
}
'.
    s := JavaScriptScanner new.
    s scanTokens:(in readStream).

    |s in|

    in := '
function scalefactor(value) {
    return "hello";
}
'.
    s := JavaScriptScanner new.
    s scanTokens:(in readStream).

    |s in|

    in := '
function scalefactor(value) {
    return /hello/;
}
'.
    in := in readStream.
    s := JavaScriptScanner for:in.
    [in atEnd] whileFalse:[
        Transcript showCR:s nextToken
    ]

    |s in|

    in := '
function scalefactor(value) {
    return a /hello/;
}
'.
    in := in readStream.
    s := JavaScriptScanner for:in.
    [in atEnd] whileFalse:[
        Transcript showCR:s nextToken
    ]