Discussion:
copying/slicing ctypes arrays, (c_ulong *n)()
(too old to reply)
RJ
2004-12-15 04:52:15 UTC
Permalink
I have some code that gets a data array from a Win32 A/D dll/driver call at 250,000 samples/sec in 2K chunks.
It has the form:
buf = (c_ulong * 2000)()
which I then copy into a circular numarray buffer like:
for i in range(2000): narray[pntr+i] = buf[i]
which is a bit slow, but fine for much lower data rates.

So, what is the fastest way copy the data? I was re-reading the Python speed FAQ - I'll try
narray[pntr:pntr+2000] = map(None, buf)
tomorrow, but is that right?

Also, since narray is circular I sometimes need to split the ctypes buf in the copy when the pointer starts over, and ctypes does not seem to support slicing. So, I copy to a temp numarray and slice that, which seems sub-optimal.

The only "problem" here is that at 250Ksmps/sec it does not leave a lot of processor time to do other things.

Would it be best to use a large ctypes array instead of numarray as the circular buffer? I would still need to slice it though, with stepping, to do FFTs etc.

Thanks,
Ray


Secret anti-spam filter-passing text. Include with reply:
qwertyuiop
Armin Steinhoff
2004-12-15 13:33:43 UTC
Permalink
Hi Ray,
Post by RJ
Would it be best to use a large ctypes array instead of numarray as the circular buffer?
Yes ... because you could then use the memmove() call of libc in order to copy .

Regards

Armin

__________________________________________________________
Mit WEB.DE FreePhone mit hoechster Qualitaet ab 0 Ct./Min.
weltweit telefonieren! http://freephone.web.de/?mc=021201
Thomas Heller
2004-12-15 17:31:21 UTC
Permalink
Post by RJ
I have some code that gets a data array from a Win32 A/D dll/driver
call at 250,000 samples/sec in 2K chunks.
buf = (c_ulong * 2000)()
for i in range(2000): narray[pntr+i] = buf[i]
which is a bit slow, but fine for much lower data rates.
So, what is the fastest way copy the data? I was re-reading the
Python speed FAQ - I'll try
narray[pntr:pntr+2000] = map(None, buf)
tomorrow, but is that right?
Should work, but builds a large temporary list. Best would be to avoid
temporary objects at all, like this (for memmove, you need the latest
ctypes release) (I assume that it is possible to get the address of a
numarray buffer in a similar way as is possible for a normal array
instance):

from ctypes import *
import array

src = array.array("i", range(32))
dst = (c_int * 32)()

memmove(dst, *src.buffer_info())

Both ctypes instances and array instances support the buffer interface,
but I don't know of a way to use that from Python code (except for the
readinto() method of file objects).
Post by RJ
Also, since narray is circular I sometimes need to split the ctypes
buf in the copy when the pointer starts over, and ctypes does not seem
to support slicing. So, I copy to a temp numarray and slice that,
which seems sub-optimal.
Currently, ctypes only supports slicing for reading from array and
pointer objects, the next release will probably support writing slices
too (although that's much more dangerous).
Post by RJ
The only "problem" here is that at 250Ksmps/sec it does not leave a
lot of processor time to do other things.
Would it be best to use a large ctypes array instead of numarray as
the circular buffer? I would still need to slice it though, with
stepping, to do FFTs etc.
Thanks,
Ray
qwertyuiop
Thomas
Ray Schumacher
2004-12-15 18:57:45 UTC
Permalink
Post by Thomas Heller
Should work, but builds a large temporary list. Best would be to avoid
temporary objects at all, like this (for memmove, you need the latest
ctypes release) (I assume that it is possible to get the address of a
numarray buffer in a similar way as is possible for a normal array
I'm not sure it will be able to work...
The closest thing to .buffer_info() for numarray-addressing seems to be
Post by Thomas Heller
Post by RJ
src = numarray.array(range(32), Float)
s = info(src)
class: <class 'numarray.numarraycore.NumArray'>
shape: (32,)
strides: (8,)
byteoffset: 0
bytestride: 8
itemsize: 8
aligned: 1
contiguous: 1
data: <memory at 00815880 with size:256 held by object 00815860 aliasing object
00000000>
byteorder: little
byteswap: 0
type: Float64
Post by Thomas Heller
Post by RJ
src._byteoffset
0

The def from numarraycore.py:

def info(self):
"""info() prints out the key attributes of a numarray."""
_gen.NDArray.info(self)
print "byteorder:", self._byteorder
print "byteswap:", self.isbyteswapped()
print "type:", repr(self._type)
Post by Thomas Heller
Post by RJ
dir(src) does not reveal anything else useable...
from ctypes import *
import array
src = array.array("i", range(32))
dst = (c_int * 32)()
memmove(dst, *src.buffer_info())
Both ctypes instances and array instances support the buffer interface,
but I don't know of a way to use that from Python code (except for the
readinto() method of file objects).
Post by RJ
Also, since narray is circular I sometimes need to split the ctypes
buf in the copy when the pointer starts over, and ctypes does not seem
to support slicing. So, I copy to a temp numarray and slice that,
which seems sub-optimal.
Currently, ctypes only supports slicing for reading from array and
pointer objects, the next release will probably support writing slices
too (although that's much more dangerous).
I am going to try re-implementing with array instead of numarray and see
how it goes. FFT can digest array.array()s as input anyway.
Since I still need a circular buffer to keep memory/storage use from
running away I'll make the array ~10% longer than needed and just track the
"end" as another variable, endPntr, setting pntr=0 anytime the actual data
saved extends past 100000. Sort of an artificial variable length array.

I'll let you know how the speed goes.

Thanks again Thomas,

Ray
Thomas Heller
2004-12-15 19:25:33 UTC
Permalink
Post by Ray Schumacher
Post by Thomas Heller
Should work, but builds a large temporary list. Best would be to avoid
temporary objects at all, like this (for memmove, you need the latest
ctypes release) (I assume that it is possible to get the address of a
numarray buffer in a similar way as is possible for a normal array
I'm not sure it will be able to work...
The closest thing to .buffer_info() for numarray-addressing seems to
Post by Thomas Heller
src = numarray.array(range(32), Float)
s = info(src)
class: <class 'numarray.numarraycore.NumArray'>
shape: (32,)
strides: (8,)
byteoffset: 0
bytestride: 8
itemsize: 8
aligned: 1
contiguous: 1
data: <memory at 00815880 with size:256 held by object 00815860 aliasing object
00000000>
byteorder: little
byteswap: 0
type: Float64
Post by Thomas Heller
src._byteoffset
0
What a pity. (You could try to parse the <memory at 00815880> to get the
address).

Maybe you should ask to make sure that there's no way to copy between
objects implementing the buffer protocol with some Python function that
I do not know about?

If such a function doesn't exist, ctypes' memmove function should be
changed to accept objects implementing the buffer interface.

Thomas
Florian Schulze
2004-12-16 08:55:09 UTC
Permalink
Post by Ray Schumacher
Post by Thomas Heller
Should work, but builds a large temporary list. Best would be to avoid
temporary objects at all, like this (for memmove, you need the latest
ctypes release) (I assume that it is possible to get the address of a
numarray buffer in a similar way as is possible for a normal array
I'm not sure it will be able to work...
The closest thing to .buffer_info() for numarray-addressing seems to be
Try src._data (where src is a numarray.ArrayType, I don't know about
Numeric, but it should be similar), it's a Memory object which should be
able to function as a buffer.

Regards,
Florian Schulze
Florian Schulze
2004-12-16 09:24:34 UTC
Permalink
On Thu, 16 Dec 2004 09:55:09 +0100, Florian Schulze
Post by Florian Schulze
Post by Ray Schumacher
Post by Thomas Heller
Should work, but builds a large temporary list. Best would be to avoid
temporary objects at all, like this (for memmove, you need the latest
ctypes release) (I assume that it is possible to get the address of a
numarray buffer in a similar way as is possible for a normal array
I'm not sure it will be able to work...
The closest thing to .buffer_info() for numarray-addressing seems to be
Try src._data (where src is a numarray.ArrayType, I don't know about
Numeric, but it should be similar), it's a Memory object which should be
able to function as a buffer.
Regards,
Florian Schulze
I posted some experiments on the numeric list, Ray asked the same question
there and that forced me to try it out ;)

Regards,
Florian Schulze
RayS
2004-12-16 15:49:55 UTC
Permalink
Thanks Florian,

My preliminary assessment from trials yesterday (based only on a rough benchmark of idle time) is that
for i in range(2000): numArray[pntr+i] = buf[i]
is >10x slower than
numArray[pntr:pntr+2000] = map(None, buf)
which is 25% slower than using array.array() & memmove
memmove(array.buffer_info()[0]+pntr, buf)

I'll test the direct assignment today, as below, as subsequent operations on numarray should be faster than array. I don't recall why I didn't.

The numarray buffer interface is interesting, but I mainly need to do a (deep) copy out of the ctypes memory.
Since the main issue is that src (the ctypes obj) is being updated with 2KB A/D data @125x per second (250K samp./s), I need to get it out of the cytpes obj into storage as efficiently as possible. That storage must then work with slicing and FFT.

If the ctypes construct buf = (c_ulong * 2000)() supported slicing, I would just leave it there...

The small buffer_read thread I made has a time.sleep(.001) for when the A/D buffer is not ready, but even at .001s, it only gets hit once per cycle, if at all. It also seemed that sleep(.0001)==sleep(0); too bad.

Thanks,
Ray
I'm posting here to see if numarray has a method that could work like array.buffer_info(). numarray.info() returns a text output that can't be used like the array method is to memmove() between ctypes and Python arrays without parsing, apparently.
ctypes thread with Thomas Heller below. His main question: "Maybe you should ask to make sure that there's no way to copy between
objects implementing the buffer protocol with some Python function that
I do not know about?"
import ctypes
a = (ctypes.c_int * 5)()
a[0] = 1; a[1] = 2; a[2] = 3; a[3] = 4; a[4] = 5
list(a)
[1, 2, 3, 4, 5]
import numarray
buf = numarray.zeros(shape=20, type='i4')
buf
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
buf[2:7] = a
buf
array([0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
temp = numarray.array(sequence=buffer(a), shape=5, type='i4')
temp
array([1, 2, 3, 4, 5])
temp._data
<read-only buffer for 0x00FA9230, ptr 0x00BFA668, size 20 at 0x010D6DA0>
buffer(a)
<read-only buffer for 0x00FA9230, ptr 0x00BFA668, size 20 at 0x00FAF600>
a[2] = 10
temp
array([ 1, 2, 10, 4, 5])
The first block just creates the ctypes data.
The second block uses slice assignment to copy the data from the ctypes array into the numarray.array.
The third block uses the buffer interface to create a numarray.array which points to the same memory location as the ctypes array.
The forth block illustrates that it's really the same memory.
You have to benchmark which one of the two solutions is the better one for you.
Regards,
Florian Schulze
______________________________________________
Numpy-discussion mailing list
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Thomas Heller
2004-12-16 16:10:10 UTC
Permalink
Post by RayS
Thanks Florian,
My preliminary assessment from trials yesterday (based only on a rough
benchmark of idle time) is that
for i in range(2000): numArray[pntr+i] = buf[i]
is >10x slower than
numArray[pntr:pntr+2000] = map(None, buf)
which is 25% slower than using array.array() & memmove
memmove(array.buffer_info()[0]+pntr, buf)
Do I understand it correctly that memmove is only slightly faster than
the 'numArray[..] = map(None, buf)' call? If so, interesting.
Post by RayS
I'll test the direct assignment today, as below, as subsequent
operations on numarray should be faster than array. I don't recall why
I didn't.
The numarray buffer interface is interesting, but I mainly need to do
a (deep) copy out of the ctypes memory. Since the main issue is that
second (250K samp./s), I need to get it out of the cytpes obj into
storage as efficiently as possible. That storage must then work with
slicing and FFT.
If the ctypes construct buf = (c_ulong * 2000)() supported slicing, I
would just leave it there...
But it *should* support slicing. Can't you write this instead of the map() call?

numArray[pntr:pntr+2000] = buf[0:2000]

Another possibility would be to let the ADC write the data directly into
the numArray, once you know it's address (and assuming it doesn't
change).

Thomas
Florian Schulze
2004-12-16 17:07:24 UTC
Permalink
Post by Thomas Heller
Another possibility would be to let the ADC write the data directly into
the numArray, once you know it's address (and assuming it doesn't
change).
AFAIKT it won't change. One feature of numarray is mmap support and for
this the memory must stay where it is.
Post by Thomas Heller
Thomas
Regards,
Florian Schulze
Ray S
2004-12-16 21:24:21 UTC
Permalink
Hi Thomas,

I made a better direct comparison of ctypes and memmove, map etc., using
all the methods suggested so far. I think it's fairly valid.(?) Run on a
2.4GHz WinXP, Py2.3, a number of times.

Parsing numarray,info() was a royal pain, it writes directly to stdout!
Post by RayS
inf = src.info()
class: <class 'numarray.numarraycore.NumArray'>
shape: (32,)
strides: (8,)
byteoffset: 0
bytestride: 8
itemsize: 8
aligned: 1
contiguous: 1
data: <memory at 00815950 with size:256 held by object 00815930 aliasing object
00000000>
byteorder: little
byteswap: 0
type: Float64
Post by RayS
inf
type(inf)
<type 'NoneType'>


I was surprised that assignment wasn't faster and that numarray assignment
was consistently ~.5% faster than Numeric.
The N vs. n memmove() flip-flopped for fastest, with array.array always slower.

Since the ctype does support slicing, I was considering leaving the data in
the device driver buffer (~1MB circluar) and poking into it, but memmove is
so much faster than slicing the ctype doing memmove()s to numarray.
I presume that I should check for numarray.iscontiguous( ) or is_c_array(
) first to be safe...

Results and code below.

Thanks to all for the help,
Ray
python test.py
Array address 11443208
n address 17039424
N address 21102656
for loop 2027.0 us ea
map 1781.0 us ea
slice 1704.0 us ea
assign N 1244.0 us ea
assign n 1242.0 us ea
memmove 4.3831 us ea
memmove N 3.4773 us ea
memmove n 3.4803 us ea

_____________________________________________________

## test.py
import array
import numarray
import ctypes
import time
import string
import StringIO
import sys

buf = (ctypes.c_long * 2000)()
Array = array.array("l", [0]*10000)
n = numarray.zeros((1000000), numarray.Int32)
N = numarray.zeros((1000000), numarray.Int32)

#!!!!!!!!!!!!!!! arrrrgggg! !!!!!!!!!!!!!!!!
# n.info() writes directly to stdout!
stdout = sys.stdout
fileo = StringIO.StringIO()
sys.stdout = fileo
n.info()
ninfo = fileo.getvalue( )
fileo.close()
sys.stdout = stdout

stdout = sys.stdout
fileo = StringIO.StringIO()
sys.stdout = fileo
N.info()
Ninfo = fileo.getvalue( )
fileo.close()
sys.stdout = stdout
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

print 'Array address', Array.buffer_info()[0]

ninfo = string.split(ninfo)
nAddress = int(ninfo[20], 16)
print 'n address', nAddress
Ninfo = string.split(Ninfo)
NAddress = int(Ninfo[20], 16)
print 'N address', NAddress

t0 = time.clock()
for loop in range(1000):
for i in range(2000):
n[loop+i] = buf[i]
print 'for loop ', round((time.clock()-t0)*1000), 'us ea'

t0 = time.clock()
for loop in range(1000):
n[loop:loop+2000] = map(None, buf)
print 'map ', round((time.clock()-t0)*1000), 'us ea'

t0 = time.clock()
for loop in range(1000):
n[loop:loop+2000] = buf[0:2000]
print 'slice ', round((time.clock()-t0)*1000), 'us ea'

t0 = time.clock()
for loop in range(10000):
N[loop:loop+2000] = buf
print 'assign N', round((time.clock()-t0)*100), 'us ea'

t0 = time.clock()
for loop in range(10000):
n[loop:loop+2000] = buf
print 'assign n', round((time.clock()-t0)*100), 'us ea'

t0 = time.clock()
for loop in range(10000):
ctypes.memmove(10+Array.buffer_info()[0],
buf,
2000)
print 'memmove ', round((time.clock()-t0)*1, 4), 'us ea'

t0 = time.clock()
for loop in range(10000):
ctypes.memmove(10+NAddress,
buf,
2000)
print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea'

t0 = time.clock()
for loop in range(10000):
ctypes.memmove(10+nAddress,
buf,
2000)
print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea'
Post by RayS
Thanks Florian,
My preliminary assessment from trials yesterday (based only on a rough
benchmark of idle time) is that
for i in range(2000): numArray[pntr+i] = buf[i]
is >10x slower than
numArray[pntr:pntr+2000] = map(None, buf)
which is 25% slower than using array.array() & memmove
memmove(array.buffer_info()[0]+pntr, buf)
Do I understand it correctly that memmove is only slightly faster than
the 'numArray[..] = map(None, buf)' call? If so, interesting.
Post by RayS
I'll test the direct assignment today, as below, as subsequent
operations on numarray should be faster than array. I don't recall why
I didn't.
The numarray buffer interface is interesting, but I mainly need to do
a (deep) copy out of the ctypes memory. Since the main issue is that
second (250K samp./s), I need to get it out of the cytpes obj into
storage as efficiently as possible. That storage must then work with
slicing and FFT.
If the ctypes construct buf = (c_ulong * 2000)() supported slicing, I
would just leave it there...
But it *should* support slicing. Can't you write this instead of the
map() call?
numArray[pntr:pntr+2000] = buf[0:2000]
Another possibility would be to let the ADC write the data directly into
the numArray, once you know it's address (and assuming it doesn't
change).
Thomas
Ray S
2004-12-17 19:48:42 UTC
Permalink
New results, corrected code...
python test.py
Array address: 11784976
n address: 17039424 <memory at 01040040 with size:4000000 held by object
01040020 aliasing object 00000000>
N address: 11384616 <built-in method __copy__ of array object at 0x00ADB6D8>

for loop 1929.0 us ea
map 1630.0 us ea
slice 1589.0 us ea
assign n 1164.0 us ea
assign N 337.0 us ea
memmove A 3.9847 us ea
memmove n 3.2005 us ea

I could not do a memmove() with Numeric because I couldn't find a
corresponding output to repr(n._data) (thanks for that tip);
repr(N.__copy__) gives the address of an object(array or method?). dir(N)
and dir(Numeric) were no help. In test.py I tried adding 40 hex (al-la
numarray) to get the first data element, then I tried 80.
Ninfo = string.split(repr(N.__copy__))
NAddress = int(Ninfo[7][:-1], 16)+int(hex(80), 16)
Adding 40 to the address of the object apparently overwrites the object, as
a GC error is thrown. Adding 80 allows the loop to run, but MS Win2K
complains that python23.dll had a problem and wants to send an error
report. I might be overwriting the __copy__ method in the dll for all I
know - the error makes that likely.

In any case, Numeric assignment is much faster than numarray.

I'm not sure why a memmove to Array is always slower than numarray.

I'm using memmove to numarray in any case; I'll post a code snip later.

The test_stub.py just starts the test with
win32process.REALTIME_PRIORITY_CLASS for consistency.

Thanks for all your input,
Ray
Thomas Heller
2004-12-21 18:50:24 UTC
Permalink
Post by Ray S
New results, corrected code...
python test.py
Array address: 11784976
n address: 17039424 <memory at 01040040 with size:4000000 held by object
01040020 aliasing object 00000000>
N address: 11384616 <built-in method __copy__ of array object at 0x00ADB6D8>
for loop 1929.0 us ea
map 1630.0 us ea
slice 1589.0 us ea
assign n 1164.0 us ea
assign N 337.0 us ea
memmove A 3.9847 us ea
memmove n 3.2005 us ea
The memmove results are probably still somewhat misleading, because of
Post by Ray S
I'm not sure why a memmove to Array is always slower than numarray.
I could not do a memmove() with Numeric because I couldn't find a
corresponding output to repr(n._data) (thanks for that tip);
repr(N.__copy__) gives the address of an object(array or method?).
dir(N) and dir(Numeric) were no help. In test.py I tried adding 40
hex (al-la numarray) to get the first data element, then I tried 80.
Ninfo = string.split(repr(N.__copy__))
NAddress = int(Ninfo[7][:-1], 16)+int(hex(80), 16)
Adding 40 to the address of the object apparently overwrites the
object, as a GC error is thrown. Adding 80 allows the loop to run, but
MS Win2K complains that python23.dll had a problem and wants to send
an error report. I might be overwriting the __copy__ method in the dll
for all I know - the error makes that likely.
I have thought about all this a little bit, and it seems that it would
be very useful to accept Python objects which implement the buffer
interface as function parameters (currently only ctype instances,
strings and integers are allowed).

Of course, sometimes you don't want the data to be copied just at the
beginning, but in this case you should wrap the object into a buffer
object (with the buildin buffer() type/function). The buffer() call
accepts optional parameters for offset and size, so address arithmetic
would be done by the buffer object itself - nicer than to add more
parameters to the memmove function and friends.

How does that sound?

Thomas
Florian Schulze
2004-12-22 20:55:29 UTC
Permalink
Post by Thomas Heller
I have thought about all this a little bit, and it seems that it would
be very useful to accept Python objects which implement the buffer
interface as function parameters (currently only ctype instances,
strings and integers are allowed).
Of course, sometimes you don't want the data to be copied just at the
beginning, but in this case you should wrap the object into a buffer
object (with the buildin buffer() type/function). The buffer() call
accepts optional parameters for offset and size, so address arithmetic
would be done by the buffer object itself - nicer than to add more
parameters to the memmove function and friends.
How does that sound?
Sounds good.

I wonder what the python developers plan to do with buffer in the future,
I read it's deprecated and should be redone.

Regards,
Florian Schulze
Thomas Heller
2004-12-23 15:58:59 UTC
Permalink
Post by Florian Schulze
Post by Thomas Heller
I have thought about all this a little bit, and it seems that it would
be very useful to accept Python objects which implement the buffer
interface as function parameters (currently only ctype instances,
strings and integers are allowed).
Of course, sometimes you don't want the data to be copied just at the
beginning, but in this case you should wrap the object into a buffer
object (with the buildin buffer() type/function). The buffer() call
accepts optional parameters for offset and size, so address arithmetic
would be done by the buffer object itself - nicer than to add more
parameters to the memmove function and friends.
How does that sound?
Sounds good.
I wonder what the python developers plan to do with buffer in the
future, I read it's deprecated and should be redone.
The buffer interface and the buffer object are separate things, the
latter is not loved very much.

There have been changes in Python 2.4 which make buffer objects
somewhat safer. The address of the underlying object is no longer
stored in the buffer object anymore, it's only retrieved when needed.
But I don't have the energy to start another buffer discussion on
python-dev right now, and since ctypes is dangerous anyway, the buffer
object is fine imo.

Thomas
RayS
2004-12-23 00:42:59 UTC
Permalink
Thomas Heller
2004-12-23 15:56:04 UTC
Permalink
Hi,
Post by Thomas Heller
But it *should* support slicing.
Yes, I got it going correctly, again...
Post by Thomas Heller
Another possibility would be to let the ADC write the data directly into
the numArray, once you know it's address (and assuming it doesn't
change).
I'm trying to figure out how this would be accomplished; I currently
create a (c_long*2000)() to receive the result of the driver call
buf = usb.EDRE_ADGetDataRaw(Sn,*Buf ,*BufSize)
where
usb = windll.EDRAPI #driver
Sn 32-bit unsigned integer Board’s serial number.
*Buf Pointer to an array of 16-bit unsigned words
*BufSize Pointer to an array of 32-bit unsigned integers
As per you suggestion, the address and memmove() work fine, the
address calcs are easy. So, how can I set up a windll call to the
driver to plop data directly into the numarray?
Pass the integer address as second parameter.

Thomas
RayS
2005-02-01 16:38:28 UTC
Permalink
Post by Thomas Heller
So, how can I set up a windll call to the
driver to plop data directly into the numarray?
Pass the integer address as second parameter.
Thanks Thomas, I used that with numarray without apparent ill effects, but Todd Miller has also recently suggested that the pointer might not be reliable (which only means that it needs to be re-read each time).

On a related topic, I asked over in Numpy-discussion
http://sourceforge.net/mailarchive/forum.php?thread_id=6468875&forum_id=4890
if there was a similar possibility directly with Numeric (we had discussed this a bit here before
http://sourceforge.net/mailarchive/message.php?msg_id=10368635)
since it is faster for small-ish arrays.
Part of it boils down to: could I use ctypes' pointer(offset) to read the Numeric structure's pointer-to-data[0]-value and then use it in memmove()?
Is this reasonable?

Alternatively, do you expect to implement the acceptance of Python objects which implement the buffer interface in the near future?

Ray


My post to Numpy-discussion:
##################################################
Thanks Todd, Travis,

Yes, Travis, I _was_ using ctypes' memmove() with numarray to great benefit. I had also understood that the address of the numarray data was stable.

Originally, I was trying to move data from a hardware A/D driver DLL call to Python as quickly as possible and did some benchmarking, so as per Thomas Heller's suggestion on the ctypes list I used memmove():
http://sourceforge.net/mailarchive/forum.php?thread_id=6166311&forum_id=24606
from ctypes import *
import array
src = array.array("i", range(32))
dst = (c_int * 32)()
memmove(dst, *src.buffer_info())
memmove() is orders of magnitude faster than map() or assignments.
I then tested and found numarray faster for the rest of the tasks (FFT etc.) http://sourceforge.net/mailarchive/forum.php?thread_id=6205658&forum_id=24606
and so parsed the info() output and plugged the data[0] address into the call.

Thomas Heller (in the above) suggested he make a mod to ctypes to accept Python objects which implement the buffer interface as function parameters, which would allow Numeric use once implemented.
I like numarray's breadth of methods so I used it at the time, but in another try at speed-squeezing yesterday afternoon I found Numeric's FFT and subtraction to be >30% faster in this case, so I switched that code over (this increase includes the use of map() with Numeric to read the A/D in another thread). Using memmove() with Numeric would speed up the reader thread once again.
Post by Thomas Heller
What is the actual address of the first element?
In C, look at a->data.
I had read the Numeric API and looked at the PyObject structure, as Travis then suggested, but my question then is: if the offset from the "array object at 0x..." (object address value) to the array[0] address is not fixed and must be read from the pointer in the PyObject structure, can we get that pointer's value directly from Python or ctypes?
ctypes pointer() "allows" poking in and directly reading memory:
"It is also possible to use indexes different from 0, but you must know what you're doing when you use this: You access or change arbitrary memory locations when you do this."
http://starship.python.net/crew/theller/ctypes/tutorial.html
So, could I use ctypes' pointer(offset) to read the structure's pointer-to-data[0]-value and then use it in memmove()?
I'll cross-post this to ctypes-users...
Post by Thomas Heller
I think the "fragile data pointer" is generally useful information, but
not completely dependable so I gave it the garish name it has.
Comments?
It's quite reasonable.
Is the data pointer value really often changed for contiguous arrays in small stand-alone Python apps? If so, I may stick with map() to avoid having to re-read the pointer before each data buffer transfer.

If anyone is interested I could post some relevant code snips from the office...

Thanks for the help,
Ray
Thomas Heller
2005-02-02 21:09:23 UTC
Permalink
Post by RayS
Alternatively, do you expect to implement the acceptance of Python
objects which implement the buffer interface in the near future?
I hope to be able to make a new release this or next week. So far, I
think objects implementing the buffer interface should *not* be allowed
as function parameters, but *buffer objects* should be. The reason is
that the former would break too much code.

IIUC, you should then be able to call buffer() on a Numeric array
instance and pass that to a function, or call buffer() on the _data
member of a Numarray array instance.

The buffer() object has been changed in Python 2.4 (or was it even 2.3?)
so that it retrieves the memory block address from the original object
each time it is needed, instead of getting it once and then storing it
permanently. So it should be even safe if the original object changes
its internal memory block.

Thomas
Ray Schumacher
2005-02-02 22:24:34 UTC
Permalink
Post by Thomas Heller
Post by RayS
Alternatively, do you expect to implement the acceptance of Python
objects which implement the buffer interface in the near future?
I hope to be able to make a new release this or next week. So far, I
think objects implementing the buffer interface should *not* be allowed
as function parameters, but *buffer objects* should be. The reason is
that the former would break too much code.
IIUC, you should then be able to call buffer() on a Numeric array
instance and pass that to a function, or call buffer() on the _data
member of a Numarray array instance.
The buffer() object has been changed in Python 2.4 (or was it even 2.3?)
so that it retrieves the memory block address from the original object
each time it is needed, instead of getting it once and then storing it
permanently. So it should be even safe if the original object changes
its internal memory block.
Ahh, thanks for the insight.
Post by Thomas Heller
Post by RayS
import Numeric, string
N=Numeric.zeros((100,), Numeric.Float)
repr(N.__copy__)
'<built-in method __copy__ of array object at 0x007B10B8>'
Post by Thomas Heller
Post by RayS
buffer(N, 0, 0)
<read-only buffer for 0x007B10B8, ptr 0x00809A98, size 0 at 0x0082B0D0>
Post by Thomas Heller
Post by RayS
buffer(N, 50, 50)
<read-only buffer for 0x007B10B8, ptr 0x00809ACA, size 50 at 0x00828E98>
Post by Thomas Heller
Post by RayS
string.split(buf.__repr__())[5]
'0x00809A98,'
Post by Thomas Heller
Post by RayS
int(string.split(buf.__repr__())[5][:-1], 16)
8428184

I presume (!) that the ptr address could be passed to ctypes.memmove()
At least, I'll try; memmove() does work fine with numarray.info address.

The desire is to write form the ctypes DLL call into the Numeric array, so
actually using the buffer(N) is out.

Ray
Thomas Heller
2005-02-03 18:32:50 UTC
Permalink
Post by Ray Schumacher
import Numeric, string
N=Numeric.zeros((100,), Numeric.Float)
repr(N.__copy__)
'<built-in method __copy__ of array object at 0x007B10B8>'
buffer(N, 0, 0)
<read-only buffer for 0x007B10B8, ptr 0x00809A98, size 0 at 0x0082B0D0>
buffer(N, 50, 50)
<read-only buffer for 0x007B10B8, ptr 0x00809ACA, size 50 at 0x00828E98>
string.split(buf.__repr__())[5]
'0x00809A98,'
int(string.split(buf.__repr__())[5][:-1], 16)
8428184
I presume (!) that the ptr address could be passed to ctypes.memmove()
At least, I'll try; memmove() does work fine with numarray.info address.
The desire is to write form the ctypes DLL call into the Numeric
array, so actually using the buffer(N) is out.
Ray, it seems I don't understand you. Why should buffer() not work?
Don't be confused that the buffer() object says <read-only buffer ...>!
The buffer call only asks for readable memory..., but ctypes doesn't
care about the readonly attribute - it will happily write into this
memory.

If this is too confusing, and this may well be, ctypes could expose a
memory() function which would insist on read-write memory, but apart
from that do the same that buffer does:

c:\sf\ctypes\sandbox\tools\codegen>py23
Python 2.3.5c1 (#61, Jan 25 2005, 19:52:06) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# readline available, tab: complete
Post by Ray Schumacher
from Numeric import *
import ctypes
from _ctypes import memory
n = zeros(3, Float)
n
array([ 0., 0., 0.])
Post by Ray Schumacher
n.__copy__
<built-in method __copy__ of array object at 0x00B2C890>
Post by Ray Schumacher
memory(n)
<read-write buffer for 0x00B2C890, ptr 0x00869EF0, size 24 at 0x00B1FC00>
Post by Ray Schumacher
buffer(n)
<read-only buffer for 0x00B2C890, ptr 0x00869EF0, size 24 at 0x00B1FAA0>
Post by Ray Schumacher
memory(n, 4)
<read-write buffer for 0x00B2C890, ptr 0x00869EF4, size 20 at 0x00B2E8E0>
Thomas
Ray S
2005-02-03 20:31:10 UTC
Permalink
Post by Thomas Heller
Don't be confused that the buffer() object says <read-only buffer ...>!
The buffer call only asks for readable memory..., but ctypes doesn't
care about the readonly attribute - it will happily write into this
memory.
Hi Thomas,

Yes, I was thinking of what the shell error said upon assignment...
Post by Thomas Heller
import Numeric, ctypes, string
N = Numeric.zeros((10,), Numeric.Float)
buf = buffer(N)
buf
<read-only buffer for 0x008F9C28, ptr 0x008D7780, size 80 at 0x008FE220>
Post by Thomas Heller
int(string.split(repr(buf))[5][:-1], 16)
9271168

## numarray version
# nAddress = int(string.split(repr(N._data))[2], 16)

## Numeric version
NAddress = int(string.split(repr(buffer(N)))[5][:-1], 16)

## Load DLL here...
## do this to get data from the USB A/D's DLL
usb.GetData(usb.Sn, (bufferInsertPos * N.itemsize()) + NAddress,
ctypes.byref( (types.c_long *
buffersize)() ) )

Which is faster than getting data into a ctypes array (c_ulong *n)() and
then doing memmove() to Numeric - one less step.

Maybe this snip would be of help to some others, although more so to numpy
people.
Post by Thomas Heller
a = array.array('l',[1,2,3])
int(string.split(repr(buffer(a)))[5][:-1], 16)
8380408
Post by Thomas Heller
If this is too confusing, and this may well be, ctypes could expose a
memory() function which would insist on read-write memory, but apart
No, not confusing, just not clear to a non-expert C person that ctypes
ignores where Python is read-only. A simple note in the tutorial would be
fine.
Some over at numpy were also unaware of memmove()s' existence in the new
releases, and seemed interested.

Thanks again,
Ray

Loading...