eXept Software AG Logo

Smalltalk/X Webserver

Documentation of class 'ExternalBytes':

Home

Documentation
www.exept.de
Everywhere
for:
 [back]

Class: ExternalBytes

Inheritance:

   Object
   |
   +--Collection
      |
      +--SequenceableCollection
         |
         +--ArrayedCollection
            |
            +--UninterpretedBytes
               |
               +--ExternalBytes
                  |
                  +--Alien
                  |
                  +--ExternalBytes::ImmutableExternalBytes
                  |
                  +--ExternalInt
                  |
                  +--ExternalLong
                  |
                  +--ExternalStructure
                  |
                  +--FFIExternalValueHolder
                  |
                  +--SharedExternalBytes
                  |
                  +--UnprotectedExternalBytes
Package:
stx:libbasic
Category:
System-Support-External Memory
Version:
rev: 1.166 date: 2024/01/22 12:26:11
user: cg
file: ExternalBytes.st directory: libbasic
module: stx stc-classLibrary: libbasic

Description:


This class provides access to any memory in the system. Its main purpose
is to provide a base class for objects referencing structured external data.

Normally, instances are created by primitive code which wants to pass C-data
to Smalltalk AND grants Smalltalk access to individual bytes afterwards.

Primitives which do not want to grant this access should return instances of
ExternalAddress. See more info there. Also, have a look at ExternalFunction
which is another similar class, but specialized to represent callable C-functions.

Since the memory address of an instance stays fixed (once allocated),
it can also be used to share data with external C-parts
(which are not prepared for objects to change their address).

Use with great care - access is not always checked for out-of-bounds
or valid addresses.

Since the data is allocated outside the garbage collected smalltalk space,
its address stays fix. Thus, it can be passed to external C-functions without
any danger. However, you have to take care for freeing the memory yourself.

To help in avoiding memory bugs, instances created with #new: are
registered in a local classvar and deregistered when the underlying memory
is explicitely freed. Since a life reference (from that classvar) exists,
the garbage collector will never find these to be reclaimable, and the
underlying memory stays allocated (at the fix address) forever.
To release the memory, either #free it or #unprotect it.
The first will immediately release the memory, while the second will delay
freeing until the next garbage collect occurs.

If you need memory which is automatically freed, create
the instance via #unprotectedNew: right away. 
The underlying malloced-memory will be released as soon as no Smalltalk reference 
to the ExtBytes object exists any more 
(however, you have to know for sure, that no C-references exist to this memory).

To release all memory call #releaseAllMemory which simply sets the
AllocatedInstances class variable to nil (thus releasing those refs).

Example (automatic freeing as soon as ref to buffer is gone):
    |buffer|

    buffer := ExternalBytes unprotectedNew:100.
    ...

Example (manual freeing - never freed, if ref to buffer is gone):
    |buffer|

    buffer := ExternalBytes new:100.
    ...
    buffer free

Example (delayed automatic freeing as soon as ref to buffer is gone):
    |buffer|

    buffer := ExternalBytes new:100.
    ...
    buffer unregister

This class only supports unstructured external data
- see the companion class ExternalStructure for more.

Notice: support for external data is still being developed -
        a parser for C structure syntax and typedefs is on the way,
        making shared data with C programs much easier in the future.

Also notice, that this class may not be available or behave different
in other smalltalk systems, making code using it very unportable.
It is provided for C interfacing only.

Finally note, that ST/X's memory system is much faster than malloc/free
in the normal case - especially for short term temporary objects,
automatically reclaimed object memory is about 5-10 times faster than
malloc/free.
Things may be different for huge byte-valued objects, which are to be
reclaimed by the oldspace collector.
Anyway, for portability, we strongly warn from using this as a substitute
for byteArrays; it is meant for shared data with external C-functions ONLY.

Debugging:
    since all manual memory systems are subject of obscure errors,
    you may want to turn malloc-tracing on; this traces all allocations/frees
    done here. To do this, evaluate: 'ExternalBytes mallocTrace:true'.

    In addition, you may turn on full debugging (with 'ExternalBytes mallocDebug:true');
    if turned on, all malloc/realloc requests are remembered and later free / realloc
    requests validated against this list (i.e. to detect freeing unallocated chunks).

    To benefit from this in C-code, we recommend you use __stx_malloc() / __stx_free()
    instead of malloc() / free(). To do so, redefine them in a header file (or cc comand line)
    and recompile your external c-libraries with this.

    I used this here to find memory leaks in the Xt libraries (there are still some in
    the HTML widget ...). If mallocDebug is on, #dumpMallocChunks will print out what is
    leftOver. This may help to find trouble spots in your C-code.


copyright
COPYRIGHT (c) 1993 by Claus Gittinger
             All Rights Reserved

This software is furnished under a license and may be used
only in accordance with the terms of that license and with the
inclusion of the above copyright notice.   This software may not
be provided or otherwise made available to, or used by, any
other person.  No title to or ownership of the software is
hereby transferred.

Class protocol:

initialization
o  initialize
(comment from inherited method)
use STC optimization - do not use self

instance creation
o  address: anAddressInteger
return a new ExternalBytes object to access bytes starting at anAddressInteger.
The memory at anAddressInteger has been allocated elsewhere.
The size is not known, therefore byte accesses will NOT be checked for valid index.
Use this, if you get a pointer from some external source (such as a
C-callBack function) and you have to extract bytes from that.

DANGER ALERT: this method allows very bad things to be done to the
system - use with GREAT care (better: do not use it)

o  address: anAddressInteger size: size
return a new ExternalBytes object to access bytes starting at anAddressInteger.
The memory at anAddressInteger has been allocated elsewhere.
The size is known, which allows byte accesses to be checked for valid index.
Use this, if you get a pointer to a structure from some external source
(such as a C-callBack function) and you have to extract things from that.
The pointer is protected from GC
i.e. ST/X will not free the heap memory, when the returned reference is no longer in use.
Be careful to avoid memory leaks, when getting malloc'd memory from an external function.

DANGER ALERT: this method allows very bad things to be done to the
system - use with GREAT care (better: do not use it)

o  new: numberOfBytes
allocate some memory usable for data;
the memory is under the control of the garbage collector
(i.e. the instance will be finalized and the malloc'd memory will be freed, if the instance goes away).
Return a corresponding ExternalBytes object or raise MallocFailure (if malloc fails).

DANGER ALERT: the memory block as allocated will be automatically freed
as soon as the reference to the returned externalBytes object
is gone (by the next garbage collect).
If the memory has been passed to a C-function which
remembers this pointer, bad things may happen ....

Usage example(s): 

     |bytes|

     bytes := ExternalBytes new:100.
     bytes wordAt:1 put:1.
     bytes doubleWordAt:3 put:16r12345678.
     bytes inspect

o  newNullTerminatedFromString: aString
allocate a null terminated string containing the chars of aString

o  newNullTerminatedFromWideString: aString
allocate a null terminated wide string containing the U16-chars of aString

o  protectedNew: numberOfBytes
allocate some memory usable for data;
the memory is safe from being finalized by the garbage collector.
(i.e. it will NOT be freed when the instance is no longer references)
Return a corresponding ExternalBytes object or raise MallocFailure (if malloc fails).

Use this, if you have to pass a block of bytes to some
external destination (such as a C function) which does not copy the
data, but instead keeps a reference to it.
For example, many functions which expect strings simply keep a ref to the passed string
- for those, an ST/X string-pointer is not the right thing to pass, since ST/X objects
may change their address.

DANGER ALERT: the memory is NOT automatically freed until it is either
MANUALLY freed (see #free) or the returned externalBytes object
is unprotected or the classes releaseAllMemory method is called.
Be aware of memory leaks when using this.
This is meant to pass data to C functions, which will free it eventually

o  unprotectedNew: numberOfBytes
allocate some memory usable for data;
the memory is under the control of the garbage collector
(i.e. the instance will be finalized and the malloc'd memory will be freed, if the instance goes away).
Return a corresponding ExternalBytes object or raise MallocFailure (if malloc fails).

DANGER ALERT: the memory block as allocated will be automatically freed
as soon as the reference to the returned externalBytes object
is gone (by the next garbage collect).
If the memory has been passed to a C-function which
remembers this pointer, bad things may happen ....

Usage example(s): 

     |bytes|

     bytes := ExternalBytes unprotectedNew:100.
     bytes wordAt:1 put:1.
     bytes doubleWordAt:3 put:16r12345678.
     bytes inspect

instance release
o  releaseAllMemory
... the next GC will get 'em

malloc debug
o  dumpMallocChunks
self dumpMallocChunks

o  freeAllMallocChunks
free all stx_malloc'd memory. Be careful, this does no validation at all;
It simply walks through all chunks and frees them unconditionally.
This may be helpful during debugging memory-leaks, to release memory which
was not correctly freed by C-code. Howeve, only memory which was allocated
by __stx_malloc() is freed here - so you better compile your primitive code with
malloc redefined to stx_malloc.
Also, mallocDebug has to be on to do this.

o  mallocDebug: aBoolean
ExternalBytes mallocDebug:true
ExternalBytes mallocDebug:false

o  mallocStatistics
self mallocStatistics

o  mallocTrace: aBoolean
ExternalBytes mallocTrace:true
ExternalBytes mallocTrace:false

o  numberOfAllocatedChunks
self numberOfAllocatedChunks

queries
o  charTypeIsSigned
return true, if the machine's native chars are signed

Usage example(s): 

     ExternalBytes charTypeIsSigned

o  doubleAlignment
return the alignment of longs in structs and unions

Usage example(s): 

     ExternalBytes doubleAlignment

o  elementByteSize
for bit-like containers, return the number of bytes stored per element.
For pointer indexed classes, 0 is returned

o  floatAlignment
return the alignment of floats in structs and unions

Usage example(s): 

     ExternalBytes floatAlignment

o  int128Alignment
return the alignment of int128s in structs and unions;
nil if not supported by the machine

Usage example(s): 

     ExternalBytes int128Alignment

o  int64Alignment
return the alignment of int64s in structs and unions

Usage example(s): 

     ExternalBytes int64Alignment

o  isBuiltInClass
return true if this class is known by the run-time-system.
Here, true is returned.

o  longAlignment
return the alignment of longs in structs and unions

Usage example(s): 

     ExternalBytes longAlignment

o  longDoubleAlignment
return the alignment of long doubles in structs and unions

Usage example(s): 

     ExternalBytes longDoubleAlignment

o  pageSize

o  pointerAlignment
return the alignment of pointers in structs and unions

Usage example(s): 

     ExternalBytes pointerAlignment

o  sizeofDouble
return the number of bytes used by the machine's native doubles.
Notice: this is inlined by the compiler(s) as a constant

Usage example(s): 

     ExternalBytes sizeofDouble

o  sizeofEnums
return the number of bytes used by the machine's native enums.
Be aware, that this can be adjusted in some compilers via the __packed__ attribute;
So better double check...

Usage example(s): 

     ExternalBytes sizeofEnums

o  sizeofFloat
return the number of bytes used by the machine's native floats.
Notice: this is inlined by the compiler(s) as a constant

Usage example(s): 

     ExternalBytes sizeofFloat

o  sizeofInt
return the number of bytes used by the machine's native integer int type.
Notice: this is inlined by the compiler(s) as a constant

Usage example(s): 

     ExternalBytes sizeofInt

o  sizeofLong
return the number of bytes used by the machine's native longs.
Notice: this is inlined by the compiler(s) as a constant

Usage example(s): 

     ExternalBytes sizeofLong

o  sizeofLongDouble
return the number of bytes used by the machine's native longdouble.
If the machine does not support them, return nil.

Usage example(s): 

     ExternalBytes sizeofLongDouble

o  sizeofLongLong
return the number of bytes used by the machine's native longlongs.
If the machine does not support them, return nil.

Usage example(s): 

     ExternalBytes sizeofLongLong

o  sizeofNativeInt
return the number of bytes used by the machine's SmallInteger native values.
If the machine uses tagged pointers,
that is the same as the size of a pointer.

Usage example(s): 

     ExternalBytes sizeofNativeInt

o  sizeofPointer
return the number of bytes used by the machine's native pointer.
Notice: this is inlined by the compiler(s) as a constant,
therefore, queries like
'ExternalBytes sizeofPointer == 8'
cost nothing; they are compiled in as a constant
(and even conditionals are eliminated).

Usage example(s): 

     ExternalBytes sizeofPointer

o  sizeofShort
return the number of bytes used by the machine's native short

Usage example(s): 

     ExternalBytes sizeofShort

o  sizeofSize_t
return the number of bytes used by the machine's size_t type.
Typically, that is the same as the pointer size

Usage example(s): 

     ExternalBytes sizeofSize_t

Instance protocol:

Compatibility-Squeak
o  getHandle

o  handle

accessing
o  address
return the start address as an integer

o  atAllPut: byteValue
replace all elements of the collection by the argument, anObject.
Return the receiver.
Notice: This operation modifies the receiver, NOT a copy;
therefore the change may affect all others referencing the receiver.

o  basicAt: index
return the byte at index, anInteger;
Indices are 1-based, therefore
this is the byte at (address + index - 1)

o  basicAt: index put: value
set the byte at index, anInteger to value which must be 0..255.
Returns value (sigh).
Indices are 1-based, therefore
this is the byte at (address + index - 1)

o  byteAt: idx
return the byte at index, anInteger;
Indices are 1-based, therefore
this is the byte at (address + index - 1)

o  byteAt: idx put: value
set the byte at index, anInteger to value which must be 0..255.
Returns value (sigh).
Indices are 1-based, therefore
this is the byte at (address + index - 1)

o  copyCStringFromHeap
fetch a 0-terminated string from my pointed-to address

o  copyCStringFromHeapStartingAt: offset
fetch a 0-terminated string from my pointed-to address

o  copyUnicodeStringFromHeap
fetch a 0-terminated wide-string (16bit) from my pointed-to address

o  instVarAt: index
redefined to suppress direct access to my address, which is a non-object

o  stringAt: startIndex
return a string starting at index up to the 0-byte.
The index is a smalltalk index (i.e. 1-based).
Notice, that the inherited method depends on size being non-nil,
which is not the case if the receiver was created from char*
as returned by a called C-function

converting
o  asExternalAddress
return the start address as an external address

o  asExternalBytes

o  asString
speed up string conversions

Usage example(s): 

      #[16r41 16r42 16r43] asExternalBytes asString

o  beImmutable
make myself write-protected

copying
o  shallowCopy
create a copy of the receiver;
that means: malloc another memory chunk and copy the bytes from the
receiver into it.
The returned memory is under the control of the garbage collector;
i.e. the memory will be freed, when there is no longer any reference to the copy

filling & replacing
o  copyBytesFrom: start to: stop into: aCollection startingAt: repStart
copy elements from me into another collection, which must be a ByteArray-
like collection.
Notice: Indices are 1-based, therefore
the first byte copied out of me is the byte at (address + start - 1).

Notice: This operation modifies aCollection;
therefore the change may affect all others referencing it.

Usage example(s): 

     |extBytes byteArray|

     extBytes := (ExternalBytes unprotectedNew:16)
            replaceBytesFrom:1 to:16
            with:#[10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160]
            startingAt:1.
     byteArray := ByteArray new:100.
     extBytes copyBytesFrom:1 to:16 into:byteArray startingAt:10.
     byteArray inspect.

o  replaceBytesFrom: start to: stop with: aCollection startingAt: repStart
replace elements from another collection, which must be a ByteArray-
like collection.
Notice: Indices are 1-based, therefore
the first byte copied into me is to the byte at (address + start - 1).

Notice: This operation modifies the receiver, NOT a copy;
therefore the change may affect all others referencing the receiver.

Usage example(s): 

     (ExternalBytes unprotectedNew:16)
            replaceBytesFrom:1 to:8
            with:#[10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160]
            startingAt:1 

     (ExternalBytes unprotectedNew:16)
            replaceBytesFrom:3 to:10
            with:#[10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160]
            startingAt:4

     (ExternalBytes unprotectedNew:16)
            replaceBytesFrom:3 to:4
            with:#[10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160]
            startingAt:1

     (ExternalBytes unprotectedNew:16)
            replaceBytesFrom:0 to:9
            with:#[10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160]
            startingAt:1

     (ExternalBytes unprotectedNew:16)
            replaceBytesFrom:1 to:10
            with:#[10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160]
            startingAt:0

     (ExternalBytes unprotectedNew:16)
            replaceBytesFrom:1 to:8
            with:#[10 20 30 40]
            startingAt:1

o  replaceNullTerminatedFromString: aString
replace elements from aString, and add a 0-byte at the end

Usage example(s): 

     |p|
     p := ExternalBytes unprotectedNew:20.
     p replaceNullTerminatedFromString:'abcdef'.
     p inspect

finalization
o  executor
redefined to return a lightweight copy
- all we need is the memory handle and the size.

Usage example(s): 

      (ExternalBytes unprotectedNew:10) executor

o  finalizationLobby
answer the registry used for finalization.
ExternalBytes have their own Registry

o  finalize
some ExternalBytes object was finalized;
free the associated heap memory with it

freeing
o  free
free a previously allocated piece of memory - be very careful, there
are no checks done here. All dangers you usually have with malloc/free
are present here ...

Usage example(s): 

at least, we check for double freeing the same chunk

o  register
register the receiver to be automatically finalized by the GC

o  registerForFinalization
register the receiver to be automatically finalized by the GC

pointer arithmetic
o  referenceToBytesFrom: start to: stop
answer a new ExternalBytes referencing a range within the receiver.
BE CAREFUL: after the receiver has been freed, the new ExternalBytes
contents is undefined

printing & storing
o  displayOn: aGCOrStream
return a printed representation of the receiver for displaying

private-accessing
o  invalidateReference
clear the start address and size

o  setAddress: aNumberOrExternalAddress size: sz
set the start address and size

o  setSize: sz
set the size - warning: dangerous if wrong

private-allocation
o  allocateBytes: numberOfBytes
allocate (malloc) numberOfBytes; if doClear is true, the allocated memory is cleared.
Fail if already allocated.
Raise MallocFailure if malloc fails to allocate enough memory

o  allocateBytes: numberOfBytes clear: doClear
allocate (malloc) numberOfBytes; if doClear is true, the allocated memory is cleared.
Fail if already allocated.
Raise MallocFailure if malloc fails to allocate enough memory

queries
o  basicSize
we do not know how many bytes are valid

o  containsNon7BitAscii
return true, if any byte in the receiver has the 7th bit on.
This may look as a too specific operation to be put here,
put it is very helpful for UTF8 string reading (Java class reader),
to quickly determine, if UTF8 decoding is needed or not.
As most strings in a class file are in fact only containing 7bit ascii,
this should speedup class file reading considerably

Usage example(s): 

     #[1 2 3 1 2 3 1 2 127 ] asExternalBytes containsNon7BitAscii
     #[1 2 3 1 2 3 1 2 250 251 250 251 255] asExternalBytes containsNon7BitAscii

o  isNull
(comment from inherited method)
for compatibility with ExternalBytes

o  isValid
true if I have an address

o  species
when copying, or concatenating, return instances of this class

registration
o  forgetMemory
forget the underlying memory - i.e. it will NOT be freed by me,
and actually no reference to the underlying memory is kept.
Warning:
Unless freed by someone else (typically a C-program/client),
this leads to a memory leak.
Use this only, if memory which was allocated by me
is given to a C-program which frees the memory.

o  protectFromGC
arrange for the receiver to remain alive.
Enter a reference to the receiver into the AllocatedInstances
class variable - this prevents it from ever being finalized by
the garbage collector, thus protecting the underlying memory.

Usage example(s): 

using a Semaphore can cause a deadlock, since unprotectFromGC may be called by
     a finalization method

o  unprotectFromGC
reenable being collected and finalized.
Remove the receiver from the AllocatedInstances
class variable - if there is no other reference to the receiver,
and this was ever allocated by me (i.e. not by the outside world),
the next garbage collect will finalize the receiver and the underlying
memory be freed.

resizing
o  grow: numberOfBytes
reallocate (realloc) numberOfBytes.
Raise MallocFailure if realloc fails to allocate enough memory

testing
o  isExternalBytes

Private classes:

    ImmutableExternalBytes

Examples:


These examples below are not directly executable;
some require primitive code to be compiled, or are simply given as sceletton.
Notice, that in C, indexing is 0-based, while in Smalltalk, indices start
at 1.

allocating memory in ST, passing it to C:

  in Smalltalk:
    ...
    bytes := ExternalBytes new:100.
    ...

  in C (assuming that the bytes-object has been passed):

    ...
    char *bytePtr;

    bytePtr = (char *)(__externalBytesAddress(bytes));
    if (bytePtr) {
        ... do something with bytes at bytePtr ...
    }

  freeing (in ST):

    ...
    bytes free.
    ...

allocating memory in C, passing it to ST:

  in C:
    ...
    char *bytePtr;

    bytePtr = (char *)(malloc(100));
    ...
    RETURN (__MKEXTERNALBYTES(bytePtr));

  in Smalltalk (extracting bytes, integers or strings):

    byteVal := bytes byteAt:1.
    ...
    intVal := bytes doubleWordAt:1 MSB:true.
    ...
    string := bytes stringAt:20.
    ...


|data bytes2|

data := #[0 1 2 3 4 5 6 7 8 9 9 8 7 6 5 4 3 2 1 0] copy.
bytes2 := ExternalBytes new:30.
bytes2 replaceBytesFrom:1 to:20 with:data startingAt:1.
data replaceBytesFrom:2 to:20 with:bytes2 startingAt:1.
bytes2

|data bytes1 bytes2|

data := #[0 1 2 3 4 5 6 7 8 9 9 8 7 6 5 4 3 2 1 0].
bytes1 := ExternalBytes new:30.
bytes2 := ExternalBytes new:30.
bytes1 replaceBytesFrom:1 to:20 with:data startingAt:1.
bytes2 atAllPut:99.
bytes2 replaceBytesFrom:2 to:21 with:bytes1 startingAt:1.
bytes2 asByteArray

|data1 bytes1 data2|

data1 := #[0 1 2 3 4 5 6 7 8 9 9 8 7 6 5 4 3 2 1 0].
bytes1 := ExternalBytes new:30.
bytes1 replaceBytesFrom:1 to:20 with:data1 startingAt:1.

data2 := ByteArray new:30 withAll:99.
data2 replaceBytesFrom:2 to:21 with:bytes1 startingAt:1.
data2

ST/X 7.7.0.0; WebServer 1.702 at 20f6060372b9.unknown:8081; Sat, 21 Dec 2024 17:03:21 GMT