www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 4681] New: Strange access violation Mandelbug with AAs + Appender

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681

           Summary: Strange access violation Mandelbug with AAs + Appender
           Product: D
           Version: D2
          Platform: Other
        OS/Version: Windows
            Status: NEW
          Severity: critical
          Priority: P2
         Component: Phobos
        AssignedTo: nobody puremagic.com
        ReportedBy: dsimcha yahoo.com



The following code produces an access violation approximately 1 in 10 times
when run.  From observations that I haven't been able to reduce to a small test
case, it appears that memory corruption is also involved.  This bug is so
non-deterministic that I haven't the slightest clue how to debug it further. 
It appears related to the order in which the arrays are appended.  I tried
changing this to some simple deterministic things and can't reproduce this bug
w/o the random index selection.

import std.stdio, std.random, std.array;

void main() {
    Appender!(float[], float)[string] aa;

    string[] indices = ["aa", "bb", "cc", "dd", "ee", "ff", "gg", "hh", "ii"];
    foreach(i; 0..10_000) {
        float f = i;
        auto index = indices[uniform(0, indices.length)];

        if(index !in aa) {
            aa[index] = typeof(aa[index]).init;
        }
        aa[index].put(f);
    }
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 19 2010
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681


David Simcha <dsimcha yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P2                          |P1
           Severity|critical                    |regression



Can't reproduce this in 200 attempts on 2.047, whereas in 200 attempts on 2.048
I get 32 access violations.  It's a regression, and a pretty severe one.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 19 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681


kennytm gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kennytm gmail.com



Reproduced on Mac OS X (v2.048) too. The backtrace indicates the error starts
at .put().

0   x                                 0x0000253b
D3std5array18__T8AppenderTAfTfZ8Appender12readCapacityMFZk + 63
1   x                                 0x000026a9
D3std5array18__T8AppenderTAfTfZ8Appender8capacityMFZk + 81
2   x                                 0x00006372
D3std5array18__T8AppenderTAfTfZ8Appender10__T3putTfZ3putMFfZv + 118
3   x                                 0x00002330 _Dmain + 424
4   x                                 0x000118d7
D2rt6dmain24mainUiPPaZi7runMainMFZv + 23
5   x                                 0x0001180e
D2rt6dmain24mainUiPPaZi7tryExecMFMDFZvZv + 42
6   x                                 0x0001191f
D2rt6dmain24mainUiPPaZi6runAllMFZv + 59
7   x                                 0x0001180e
D2rt6dmain24mainUiPPaZi7tryExecMFMDFZvZv + 42
8   x                                 0x0001179b main + 179
9   x                                 0x0000217d start + 53

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 19 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681




A much shorter test case:

    import std.random, std.array;

    void main() {    
        Appender!(int[])[3] aa;
        foreach(i; 0..10000)
            aa[uniform(0, aa.length)].put(i);
    }

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 19 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681




It's definitely a bug in Appender, not a bug in the runtime or AAs.  Replacing
2.048 Appender with 2.047 Appender in array.d and recompiling Phobos and my
code makes this bug go away.  

All the cruft about associative arrays and stuff is just artifacts of how I
initially discovered this bug and the test case I was reducing from.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 19 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681




Deterministic test case (always crash on my machine):

----

import std.array,std.stdio;

void main() {    
    Appender!(uint[])[3] aa;

    auto x = [
       
0,1,1,0,2,1,2,2,1,2,0,0,2,1,1,1,2,0,0,0,0,0,0,2,1,2,1,0,1,2,0,0,0,2,0,1,
       
1,0,1,0,2,2,0,0,1,2,0,1,1,1,1,2,0,0,2,1,1,2,2,0,0,2,1,0,1,0,2,0,0,0,0,2,
       
2,2,1,1,0,0,2,2,0,0,2,1,2,1,0,1,0,1,2,1,1,2,1,1,0,1,2,2,1,0,1,0,0,0,2,1,
       
1,1,0,2,0,2,0,1,2,0,2,0,1,0,0,0,0,1,1,0,2,2,0,2,1,0,0,0,0,1,1,0,0,1,1,0,
       
2,2,0,2,2,2,2,0,1,1,1,2,0,0,0,2,1,2,0,2,0,1,0,2,1,0,2,1,2,0,0,1,2,1,1,2,
       
2,2,1,2,2,1,1,0,1,2,0,1,0,2,0,0,1,0,0,0,2,1,1,0,1,0,1,1,2,2,0,2,0,1,0,1,
       
2,1,1,1,0,0,0,0,1,0,2,2,2,2,2,1,0,0,0,0,0,1,0,2,1,0,2,2,1,0,2,2,0,0,1,1,
       
1,1,1,2,0,0,2,2,2,0,2,2,0,1,0,1,1,2,0,0,2,2,1,0,0,1,0,1,2,1,1,1,1,2,2,0,
       
0,2,2,2,0,2,2,2,2,1,2,1,1,2,1,0,1,1,2,1,0,2,0,1,1,2,1,0,1,2,2,0,1,2,0,0,
       
0,0,2,1,2,2,2,2,0,2,0,1,2,2,2,2,2,2,1,2,1,1,0,1,0,2,1,2,2,2,1,2,1,0,1,0,
       
2,1,2,2,2,2,2,2,0,0,1,0,2,1,0,1,0,0,1,1,0,0,2,0,0,2,2,2,2,1,1,1,0,1,0,0,
       
1,1,0,2,2,1,0,0,2,1,0,2,1,1,2,2,0,2,0,1,2,1,0,2,0,0,0,0,1,2,2,1,2,1,1,2,
       
1,1,0,1,0,2,2,2,1,1,2,2,0,2,2,2,2,2,1,1,0,0,1,0,0,0,0,0,0,1,0,2,1,1,1,1,
       
1,2,1,1,2,2,0,0,1,0,1,2,1,0,2,1,0,0,1,2,2,0,1,1,2,2,0,0,0,2,2,1,1,0,0,0,
       
0,1,2,2,0,1,0,0,1,2,0,2,2,2,1,1,0,0,0,0,1,2,0,1,0,1,1,2,2,2,2,2,0,2,1,1,
       
1,2,1,2,1,0,2,1,0,1,1,0,2,0,1,1,0,0,1,0,1,0,1,0,2,0,0,0,2,0,0,0,1,1,2,2,
       
1,0,0,1,2,2,2,0,1,0,2,2,1,2,1,1,1,1,1,2,1,1,2,1,2,1,1,1,0,0,2,0,2,2,1,1,
       
1,0,1,2,1,2,1,2,1,2,2,0,0,2,0,1,1,2,1,2,0,2,1,1,1,2,2,1,0,2,1,0,0,2,0,1,
       
2,0,0,0,1,2,2,2,1,1,2,1,2,0,0,1,2,0,2,1,1,0,1,0,1,1,2,0,0,2,0,0,1,0,2,2,
       
0,0,2,1,1,0,0,1,1,0,1,2,2,1,0,2,2,0,2,2,2,2,2,2,0,1,0,1,2,1,0,0,0,2,0,1,
       
2,2,0,2,0,2,2,0,2,1,1,2,1,0,0,1,0,1,0,1,1,0,0,2,2,2,2,0,1,0,2,1,0,0,0,1,
       
2,2,0,1,1,1,0,1,1,0,0,0,2,1,0,1,0,0,1,0,2,2,2,1,1,1,2,0,1,1,1,0,1,0,2,2,
       
1,1,0,1,0,0,2,2,2,0,2,1,2,0,0,1,0,2,0,0,1,1,1,2,1,0,1,2,2,1,2,1,0,1,2,2,
       
1,0,2,1,2,0,0,2,0,1,1,0,2,1,0,1,0,0,1,0,0,0,2,2,2,1,0,0,1,2,2,1,1,2,2,1,
       
1,1,2,2,1,2,1,1,2,1,1,0,2,2,2,0,0,2,1,2,2,1,2,0,1,1,2,0,2,0,2,1,2,2,0,1,
       
2,2,1,0,1,2,0,2,0,2,2,1,2,2,0,2,2,1,0,0,2,2,1,2,0,0,1,1,0,2,0,1,0,2,0,1,
       
1,0,0,0,2,1,1,0,0,2,1,1,0,0,2,2,2,0,1,0,0,1,2,1,2,1,2,1,2,2,2,0,1,1,1,2,
       
1,0,1,1,1,2,1,2,2,2,0,1,2,0,1,0,0,0,0,1,1,2,1,1,2,2,1,2,2,1,1,1,1,1,2,2,
       
0,0,1,1,1,1,0,2,1,0,1,0,0,2,0,0,1,0,2,1,0,1,0,0,1,2,0,1,1,1,2,1,0,1,1,2,
       
0,1,2,1,1,2,1,0,0,0,2,0,0,0,0,2,1,2,1,1,0,0,1,0,0,0,1,0,2,0,0,1,1,2,2,1,
       
2,1,2,0,0,0,0,1,0,0,1,0,0,1,0,1,0,2,0,0,0,1,1,1,1,0,2,1,2,1,1,2,2,0,1,0,
       
2,1,0,0,1,2,2,2,2,0,2,2,2,2,1,1,2,1,0,1,2,2,1,1,0,0,0,0,1,0,2,0,2,2,0,1,
       
0,0,0,2,2,0,2,0,0,2,2,2,0,1,1,1,2,1,2,1,2,2,1,0,2,1,2,1,2,1,1,0,0,1,2,1,
       
0,1,2,2,0,2,2,2,1,0,0,0,1,1,0,1,0,1,1,0,0,1,0,0,0,2,2,1,2,1,1,1,0,1,0,2,
       
0,2,1,1,0,0,1,1,2,1,2,2,0,1,2,1,1,1,2,1,1,2,1,1,0,1,0,2,0,2,2,2,0,1,1,0,
       
0,0,1,2,2,1,2,0,0,0,2,1,1,0,0,2,0,2,1,0,1,2,2,1,0,1,1,0,1,1,2,2,1,0,2,0,
       
1,2,0,0,1,2,2,2,1,1,1,2,0,2,2,0,0,0,0,0,0,2,2,2,1,2,1,0,2,0,2,1,0,0,1,0,
       
0,0,1,1,0,0,2,2,2,0,0,2,0,1,0,2,2,0,1,1,1,1,0,2,0,0,0,1,1,1,1,1,2,2,0,2,
       
2,0,1,0,2,1,0,0,1,0,0,1,1,1,0,0,0,0,2,0,1,0,1,1,0,2,2,0,0,2,0,2,1,0,1,2,
       
0,2,1,1,2,0,1,0,0,0,2,0,0,1,0,0,0,2,0,0,2,2,1,2,0,0,1,2,1,2,2,1,1,0,2,1,
       
2,2,2,1,1,1,0,2,2,2,1,2,2,2,2,0,0,2,1,0,0,1,2,1,2,1,1,2,2,1,2,1,2,0,2,1,
       
1,2,1,0,0,2,2,2,0,0,0,2,2,2,1,0,1,2,1,1,2,1,2,0,0,1,0,0,0,1,2,1,1,0,1,1,
        0,1,1
    ];

    foreach(v; x)
        aa[v].put(0);
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 19 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681





 Deterministic test case (always crash on my machine):
Apparently this test case is deterministic only on your machine. On my machine it doesn't reproduce this bug. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Aug 20 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681




Comment from Shin Fujishiro the Phobos mailing list, reproduced here to make it
more public and permanent:

Seems like a bug of Appender.writeCapacity().  It often writes 'cap' to
wrong address; the following assertion fails:
==========
diff --git phobos/std/array.d phobos/std/array.d
index 6b62733..e6d3a62 100644
--- phobos/std/array.d
+++ phobos/std/array.d
   -736,10 +736,12    private:
         auto p = cast(ubyte*) (pArray.ptr + pArray.length);
         if (cap < ubyte.max)
         {
+            assert(p + 1 <= GC.addrOf(pArray.ptr) + GC.sizeOf(pArray.ptr));
             *p = cast(ubyte) cap;
         }
         else if (cap < ushort.max)
         {
+            assert(p + 3 <= GC.addrOf(pArray.ptr) + GC.sizeOf(pArray.ptr));
             *p++ = ubyte.max;
             *p++ = cast(ubyte) cap;
             *p++ = cast(ubyte) (cap >> 8);
==========


Shin

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 20 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681


Don <clugdbug yahoo.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |clugdbug yahoo.com.au



Here's another test case for what I believe is the same thing. It's memory
corruption rather than a segfault. This is a regression since 2.047.

import std.stdio;
import std.array;

void main()
{
    Appender!(double[]) b, c;
    string[] t;
    t ~= "qqqqq";
    t ~= "qqqqq";
    double zzz;
    b.put( 111.1 );

    for (;;) {
        c.put(double.nan);
        b.put( 0);

        double qqq = b.data()[0];
        writefln("%s", qqq);
        assert(qqq>0);
    }
}
--------
111.1
111.1
111.1
111.1
nan
core.exception.AssertError test(46): Assertion failure

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 24 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681


nfxjfg gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |nfxjfg gmail.com



There's some wtfish code in Appender, that probably tries to emulate how the
runtime allocates arrays. I don't understand why it doesn't simply use the
functions exported by the runtime. Even if not, there's no justification to
duplicate all the code in Phobos, instead of extending the runtime itself.

Or maybe the original author thought a simple function call on array
reallocation is to slow?

Also I doubt using gc_realloc is a good idea: in some cases, gc_realloc frees
memory by force (similar to gc_free), and thus is unsafe in SafeD's definition.
(Although I'm not quite sure if that is the case here.)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 24 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681


Steven Schveighoffer <schveiguy yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |schveiguy yahoo.com



04:56:47 PDT ---
I checked in hopefully a fix for this.

changeset http://www.dsource.org/projects/phobos/changeset/1929

Note, I changed the interface of Appender slightly to be safer.  Instead of
taking an array reference pointer, it takes an array.  This means your original
array passed in will *not* be appended to.  To get the resulting data after
appending, use the data method.

I didn't find any cases in Phobos that were adversely affected by this change
(but I did have to change a few modules that used appender in unit tests).

I tested the new version against the original code in this bug, and Don's code
in comment 8, both no longer exhibit errors.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 26 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681




This is still full of dirty runtime calls and attempts to emulate half of
lifetime.d (though the worst part is commented).

Why doesn't it simply use the D standard way to re-allocate an array, and then
use array.capacity to see how much can be safely appended?

Something along the lines of:

private struct Data {
   T[] arr;
   size_t user_length;
}

Data _data;

void put(T item) {
   if (_data.user_length == arr.length) {
      size_t newcapacity = something larger than user_length;
      reallocate(newcapacity);
   }
   _data.arr[_data.user_length++] = item;
}

void reallocate(size_t newcapacity) {
   _data.arr.length = newcapacity;
   //include the data "overallocated" by the runtime into the array
   size_t realcap = _data.arr.capacity;
   _data.arr.length = realcap;
}

T[] data() {
   T[] arr = _data.arr[0.._data.user_length];
   _data = _data.init;
   assumeSafeAppend(arr);
   return arr;
}

Or did I overlook something.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 26 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681




05:51:15 PDT ---

 This is still full of dirty runtime calls and attempts to emulate half of
 lifetime.d (though the worst part is commented).
Why are runtime calls dirty? I don't use any undocumented runtime functions... It doesn't emulate lifetime, except for the newCapacity function (which arguably could be a call into the runtime, but it's a simple enough function, and doesn't really have to be consistent with the runtime), and the use of GC.extend. I'm notably not trying to store the length inside the memory block.
 Why doesn't it simply use the D standard way to re-allocate an array, and then
 use array.capacity to see how much can be safely appended?
Because the call to the runtime cannot be inlined, and is much slower than simply dereferencing a pointer. Appender is supposed to be as fast as possible at appending. If you want to use built-in appending, just use it. Appender is for getting the highest possible performance out of an array.
 
 Something along the lines of:
 
 private struct Data {
    T[] arr;
    size_t user_length;
 }
 
 Data _data;
 
 void put(T item) {
    if (_data.user_length == arr.length) {
       size_t newcapacity = something larger than user_length;
       reallocate(newcapacity);
    }
    _data.arr[_data.user_length++] = item;
 }
 
 void reallocate(size_t newcapacity) {
    _data.arr.length = newcapacity;
    //include the data "overallocated" by the runtime into the array
    size_t realcap = _data.arr.capacity;
    _data.arr.length = realcap;
 }
 
 T[] data() {
    T[] arr = _data.arr[0.._data.user_length];
    _data = _data.init;
    assumeSafeAppend(arr);
    return arr;
 }
 
 Or did I overlook something.
This is a different way of doing it, though I'd probably replace reallocate with a standard ~=. I think your way would work fine. I agree your way seems more congruent with the runtime. I wonder if there are any advantages to doing it that way? I think the Appender I committed is going to be slightly faster in the long run, because it avoids using runtime appending at all. However, it has some limitations in that it cannot use the rest of a passed in memory block (i.e. the space beyond the initial array). But your code gave me an idea of how to achieve that. I think this should work: this(T[] arr) { _data = new Data; _data.arr = arr; auto cap = arr.capacity; if(cap > origlen) arr.length = cap; // use up the rest of the block _data.capacity = arr.length; } I'll add this change too. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Aug 26 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681





 Why are runtime calls dirty?  I don't use any undocumented runtime functions...
Because they do more work than necessary and rely on more implementation details than necessary. Also, more bugs (oh hey look, we're posting in a bug report). I'm most worried about the assumptions of the array memory layout.
 Because the call to the runtime cannot be inlined, and is much slower than
 simply dereferencing a pointer.  Appender is supposed to be as fast as possible
 at appending.
My code only calls runtime functions when the capacity is exhausted. As long as there's enough capacity, not a single runtime function is called on appending. It's really similar to all the code that has been in Appender before, except less dirty. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Aug 26 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681




06:30:03 PDT ---


 Why are runtime calls dirty?  I don't use any undocumented runtime functions...
Because they do more work than necessary and rely on more implementation details than necessary. Also, more bugs (oh hey look, we're posting in a bug report). I'm most worried about the assumptions of the array memory layout.
I'm not assuming anything about the memory layout. GC.qalloc gives me a block of data, and I'm using the data. Its interface is well defined without any hidden assumptions. Besides, you are calling the same runtime functions, just using a different interface. I don't see how one is more "dirty" than the other, we are both using well-documented runtime functions.
 
 Because the call to the runtime cannot be inlined, and is much slower than
 simply dereferencing a pointer.  Appender is supposed to be as fast as possible
 at appending.
My code only calls runtime functions when the capacity is exhausted. As long as there's enough capacity, not a single runtime function is called on appending. It's really similar to all the code that has been in Appender before, except less dirty.
Your code calls the lifetime runtime functions, which call the GC runtime functions. My code just calls the GC functions, skipping the lifetime functions. Like your code, mine only calls runtime functions on exhaustion. BTW, the statement I was responding to asked why you can't just use runtime functions to determine capacity, I interpreted that as you wishing to use the runtime management of array memory for every append. I hadn't read your code yet. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Aug 26 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681





 I'm not assuming anything about the memory layout.  GC.qalloc gives me a block
 of data, and I'm using the data.  Its interface is well defined without any
 hidden assumptions.
That mess is mostly gone, but there's still that commented stuff. Will it be readded later?
 Besides, you are calling the same runtime functions, just using a different
 interface.  I don't see how one is more "dirty" than the other, we are both
 using well-documented runtime functions.
It doesn't zero the additional memory returned by GC.extend(). If precise GC is introduced, that won't be handled correctly either.
 Your code calls the lifetime runtime functions, which call the GC runtime
 functions.  My code just calls the GC functions, skipping the lifetime
 functions.  Like your code, mine only calls runtime functions on exhaustion.
The only real overhead is due to array initialization. If that is really so important, the runtime should provide a clean interface to allocate arrays uninitialized (or zeroed if not NO_SCAN). Basically a _d_arraysetlengthT without the initialization code. That would be useful for other code too; most time you create an array, you overwrite all its contents anyway. Keep in mind that optimizing the runtime code might be more worthwhile than optimizing Appender. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Aug 26 2010
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4681


David Simcha <dsimcha yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED


-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 18 2010