digitalmars.D - [OT] I wrote a (better?) heap memory manager, is it good for anything?
- Mehrdad (48/48) Aug 09 2014 Hi all,
- Mehrdad (8/9) Aug 09 2014 One other feature I forgot to mention:
- Anonymouse (2/51) Aug 09 2014 There is no realloc, move, cpy ?
- Mehrdad (4/5) Aug 09 2014 Not sure what you mean by move or copy (can't you just use
- Timothee Cour via Digitalmars-d (4/10) Aug 10 2014 doesn't it's not useful:
- Mehrdad (8/11) Aug 11 2014 Yeah, it's one of the shortcomings of C++ and it would be useful.
- Klaus (4/9) Aug 11 2014 Those functions are part of the memory manager (MM). Some MM have
- Era Scarecrow (13/17) Aug 11 2014 I've tried using some of them sometimes. Came across a case of
- Mehrdad (6/10) Aug 11 2014 Hmm I don't get what they have to do with the memory manager. The
Hi all, This is barely relevant to D, but I thought I'd try posting it here because I thought the audience might be interested. Let me know if I should post it somewhere else. Long story short, I wrote my own C++ heap memory manager a while ago from scratch. First, here's a description: Cons: - Average-case performance is somewhat worse than heap managers provided by the OS or language runtimes - I'm not quite ready to open-source it yet (partly because I'm not sure if it's worth licensing commercially vs. open-sourcing, and partly because I'm not quite confident it's completely bug-free yet) Pros: - It was built from the ground-up to have dependable worst-case performance, and I think it's *pretty good* and robust under various allocation patterns - It keeps metadata entirely separate from data -- so when allocate a 4KB page of memory, it's not going to allocate an extra 4KB behind it just to put a "length" field there, which has given me problems before. (This can reduce the working set of the process considerably depending on your allocation granularity.) - It is alignment-agnostic -- so, it allocates _exactly_ as much as requested (no performance penalty for this whatsoever) - It's platform-independent (it's worked on Windows and also on Linux the few times I tried it) -- it's pretty easy to supply your own memory manager underneath, too. - It has a very simple C (and C++) interface: typedef uintptr_t ManageMemory( void *context, uintptr_t address, size_t size, bool deallocate, bool commitment); extern ManageMemory manage_system_memory; void *memory_heap_construct( void *p, ManageMemory *manager, void *context); void memory_heap_destroy(void *p); void *memory_heap_allocate( void *heap, size_t size, void *hint); void memory_heap_deallocate( void *heap, void *address, size_t size); size_t memory_heap_size(); I was basically wondering if had any particular workloads they were interested in testing this on and giving me feedback on it. For now I'd give you a .DLL or a .so (though I haven't redistributed Linux binaries before, so I'd have to figure out how to do that) and the C/C++ header so you can use it and see how it is. Maybe you'd find it potentially useful for D too, but I guess that depends on how well it performs. Let me know if you're interested.
Aug 09 2014
On Saturday, 9 August 2014 at 08:48:37 UTC, Mehrdad wrote:(snip)One other feature I forgot to mention: It allows you to deallocate sub-blocks of any sizes that you wish -- this can be completely independent of the sizes they were originally allocated in, and I'm fairly certain I designed it so that there would be no performance difference if you did so. (Maybe this would be useful for a GC or some other kind of application, I don't know.)
Aug 09 2014
On Saturday, 9 August 2014 at 08:48:37 UTC, Mehrdad wrote:Hi all, This is barely relevant to D, but I thought I'd try posting it here because I thought the audience might be interested. Let me know if I should post it somewhere else. Long story short, I wrote my own C++ heap memory manager a while ago from scratch. First, here's a description: Cons: - Average-case performance is somewhat worse than heap managers provided by the OS or language runtimes - I'm not quite ready to open-source it yet (partly because I'm not sure if it's worth licensing commercially vs. open-sourcing, and partly because I'm not quite confident it's completely bug-free yet) Pros: - It was built from the ground-up to have dependable worst-case performance, and I think it's *pretty good* and robust under various allocation patterns - It keeps metadata entirely separate from data -- so when allocate a 4KB page of memory, it's not going to allocate an extra 4KB behind it just to put a "length" field there, which has given me problems before. (This can reduce the working set of the process considerably depending on your allocation granularity.) - It is alignment-agnostic -- so, it allocates _exactly_ as much as requested (no performance penalty for this whatsoever) - It's platform-independent (it's worked on Windows and also on Linux the few times I tried it) -- it's pretty easy to supply your own memory manager underneath, too. - It has a very simple C (and C++) interface: typedef uintptr_t ManageMemory( void *context, uintptr_t address, size_t size, bool deallocate, bool commitment); extern ManageMemory manage_system_memory; void *memory_heap_construct( void *p, ManageMemory *manager, void *context); void memory_heap_destroy(void *p); void *memory_heap_allocate( void *heap, size_t size, void *hint); void memory_heap_deallocate( void *heap, void *address, size_t size); size_t memory_heap_size(); I was basically wondering if had any particular workloads they were interested in testing this on and giving me feedback on it. For now I'd give you a .DLL or a .so (though I haven't redistributed Linux binaries before, so I'd have to figure out how to do that) and the C/C++ header so you can use it and see how it is. Maybe you'd find it potentially useful for D too, but I guess that depends on how well it performs. Let me know if you're interested.There is no realloc, move, cpy ?
Aug 09 2014
On Saturday, 9 August 2014 at 13:29:47 UTC, Anonymouse wrote:There is no realloc, move, cpy ?Not sure what you mean by move or copy (can't you just use regular memmove/memcpy?) but no, there is no realloc -- C++ doesn't use realloc and this was meant to be used in C++.
Aug 09 2014
On Sat, Aug 9, 2014 at 8:46 AM, Mehrdad via Digitalmars-d < digitalmars-d puremagic.com> wrote:On Saturday, 9 August 2014 at 13:29:47 UTC, Anonymouse wrote:doesn't it's not useful: http://codervil.blogspot.com/2012/11/how-to-outperform-stdvector-in-1-easy.htmlThere is no realloc, move, cpy ?Not sure what you mean by move or copy (can't you just use regular memmove/memcpy?) but no, there is no realloc -- C++ doesn't use reallocand this was meant to be used in C++.
Aug 10 2014
On Monday, 11 August 2014 at 00:52:00 UTC, Timothee Cour via Digitalmars-d wrote:doesn't it's not useful: http://codervil.blogspot.com/2012/11/how-to-outperform-stdvector-in-1-easy.htmlYeah, it's one of the shortcomings of C++ and it would be useful. That said, it'd be pretty easy for me to code it in -- there's nothing about the design that prevents me from doing so. Also, if anyone has recommendations on where else I could ask about this I'd really appreciate it. (e.g. Show HN doesn't seem particularly excited about this stuff.)and this was meant to be used in C++.
Aug 11 2014
On Saturday, 9 August 2014 at 15:46:27 UTC, Mehrdad wrote:On Saturday, 9 August 2014 at 13:29:47 UTC, Anonymouse wrote:Those functions are part of the memory manager (MM). Some MM have higly optimized memory copy operation (i.e realloc: if the source is overlapped with the dest...using SSE or MMX registers... etc).There is no realloc, move, cpy ?Not sure what you mean by move or copy (can't you just use regular memmove/memcpy?) but no, there is no realloc -- C++ doesn't use realloc and this was meant to be used in C++.
Aug 11 2014
On Monday, 11 August 2014 at 23:10:14 UTC, Klaus wrote:Those functions are part of the memory manager (MM). Some MM have higly optimized memory copy operation (i.e realloc: if the source is overlapped with the dest...using SSE or MMX registers... etc).I've tried using some of them sometimes. Came across a case of trying to do a compare or copy safely, but if it was under a certain size it just didn't do anything making my compression testing functions fail. This was mostly if they were under the system's default word size. Not sure why that was, I would have thought it would switch tracks and handle less efficient pieces if there was only a few of them so I didn't have to write special work-around cases. Of course could do like M$ does, and increase the minimum size to the word length or double that for compression (NTFS does this) and a lot of speedups appear and problems go away rather than being as thorough as possible...
Aug 11 2014
On Monday, 11 August 2014 at 23:10:14 UTC, Klaus wrote:Those functions are part of the memory manager (MM). Some MM have higly optimized memory copy operation (i.e realloc: if the source is overlapped with the dest...using SSE or MMX registers... etc).Hmm I don't get what they have to do with the memory manager. The memory manager is responsible for allocating the blocks, it doesn't actually alter the blocks (or ever need to) in any way... it's just bookkeeping. memcpy is pretty independent of the memory manager.
Aug 11 2014