View Issue Details

IDProjectCategoryView StatusLast Update
0000640LDMud 3.3Portabilitypublic2009-05-28 16:16
Reporterzesstra Assigned Tozesstra  
PrioritynormalSeverityminorReproducibilityN/A
Status resolvedResolutionfixed 
Platformx86_64OSMacOS XOS Version10.5.x
Product Version3.3.718 
Target Version3.3.719Fixed in Version3.3.719 
Summary0000640: Support mmap() for allocating memory from the system
DescriptionOur allocators usually try to use sbrk()/brk() for getting memory from the system and replace the system malloc. If this does not work, the fall back to allocate memory with malloc().
sbrk/brk() is declared legacy in SUSv2 and removed in POSIX.1-2001. Darwin has the functions, but brk() is basically a noop and sbrk() can only adjust the break value within 4MB.
Therefore we should introduce support to get memory by mmapping anonymous pages into our virtual address range. This will support REPLACE_MALLOC on Darwin/MacOS again (and maybe on other systems). When support for brk/sbrk() vanishes in mainline OS, we may remove brk() eventually.
TagsNo tags attached.

Activities

2009-05-26 17:16

 

0001-Use-mmap-for-getting-memory-from-the-system-slaba.patch (6,249 bytes)   
From 80d078af312082fcfc162b5b5f6d379083d10487 Mon Sep 17 00:00:00 2001
From: zesstra <zesstra@zesstra.de>
Date: Thu, 21 May 2009 17:22:26 +0200
Subject: [PATCH 1/3] Use mmap() for getting memory from the system (slaballoc).

sbrk() and brk() are obsoleted and some systems (like Darwin) don't have a
working implementation anymore. If sbrk/brk() are not usable, but mmap()
is available, use mmap() for mapping anonymous memory which we then use.
If mmap() is not available, use the old fallback (system malloc()).
Changes some precompiler checks for SBRK_OK, which should be REPLACE_MALLOC.

Signed-off-by: zesstra <zesstra@zesstra.de>
---
 src/slaballoc.c |   70 ++++++++++++++++++++++++++++++++++++++++++++----------
 src/xalloc.c    |    4 +-
 2 files changed, 59 insertions(+), 15 deletions(-)

diff --git a/src/slaballoc.c b/src/slaballoc.c
index 96858fd..e32f77e 100644
--- a/src/slaballoc.c
+++ b/src/slaballoc.c
@@ -25,10 +25,10 @@
  *
  *  -- Small Blocks --
  *
- * Small blocks are allocations of up to (SMALL_BLOCK_NUM+1)*4 Bytes, currently
- * 68 Bytes. Such block are allocated from larger memory blocks ('slabs')
- * sized rough multiples of 4KByte. The size is determined such that the slab
- * holds at least 100 blocks.
+ * Small blocks are allocations of up to (SMALL_BLOCK_NUM+1)*GRANULARITY Bytes,
+ * currently 68-136 Bytes. Such block are allocated from larger memory blocks 
+ * ('slabs') sized rough multiples of 4KByte. The size is determined such that
+ * the slab holds at least 100 blocks.
  *
  * A slab consists of a header, followed by the blocks themselves. Free blocks
  * within a slab are linked together in a free list. When a slab is freshly
@@ -200,13 +200,17 @@
 #include "driver.h"
 #include "typedefs.h"
 
-#ifdef HAVE_MADVISE
+#if defined(HAVE_MADVISE) || defined(HAVE_MMAP)
+#    include <sys/types.h>
 #    include <sys/mman.h>
 #    define MADVISE(new,old)  madvise(new,old,MADV_RANDOM)
 #else
 #    define MADVISE(new,old)  NOOP
 #endif
 
+// for sysconf()
+#include <unistd.h>
+
 #include "slaballoc.h"
 
 #include "mstrings.h"
@@ -246,7 +250,11 @@
 #  else
 #      undef SBRK_OK
 #  endif
-#endif
+#else if defined (HAVE_MMAP)
+#  ifdef MALLOC_SBRK
+#      define REPLACE_MALLOC
+#  endif
+#endif // SBRK_OK
 
 #define MEM_MAIN_THREADSAFE
 
@@ -344,7 +352,7 @@
     */
 
 
-#ifdef SBRK_OK
+#if defined(MALLOC_SBRK) && (defined(SBRK_OK) || defined (HAVE_MMAP))
 #    define CHUNK_SIZE          0x40000
 #else
     /* It seems to be advantagous to be just below a power of two
@@ -933,7 +941,7 @@ mem_dump_data (strbuf_t *sbuf)
                );
 
     dump_stat("\npermanent blocks:  %8lu        %10lu\n", perm_st);
-#ifdef SBRK_OK
+#ifdef REPLACE_MALLOC
     dump_stat("clib allocations:  %8lu        %10lu\n", clib_st);
 #else
     strbuf_addf(sbuf, "clib allocations:       n/a               n/a\n");
@@ -3442,7 +3450,7 @@ esbrk (word_t size, size_t * pExtra)
  *
 #ifdef SBRK_OK
  * It is system dependent how sbrk() aligns data, so we simpy use brk()
- * to insure that we have enough.
+ * to ensure that we have enough.
  * At the end of the newly expanded heap we create a fake allocated
  * block of 0 bytes so that large_malloc() knows where to stop.
 #else
@@ -3487,6 +3495,7 @@ esbrk (word_t size, size_t * pExtra)
 #else  /* not SBRK_OK */
 
     char *block;
+    size_t req_size = size; // the requested size of memory.
     word_t *p;    /* start of the fake block */
     const int overhead = TL_OVERHEAD;
     size_t overlap = 0;
@@ -3494,16 +3503,31 @@ esbrk (word_t size, size_t * pExtra)
        * the new allocation with the old one.
        */
 
-
     mdb_log_sbrk(size);
     size += overhead * GRANULARITY;  /* for the extra fake "allocated" block */
 
+    // get the new memory block
+#ifdef HAVE_MMAP
+    {
+        static size_t pagesize = 0; // pagesize - 1 for this system.
+        if (!pagesize)
+            pagesize = getpagesize() - 1;
+        // round to multiples of the real pagesize
+        if ((size & pagesize) != 0)
+        {
+            size += pagesize;
+            size &= ~pagesize;
+        }
+        // get new page(s)
+        block = mmap(0, size, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0);
+        if (block == MAP_FAILED)
+            return NULL;
+    }
+#else
     block = malloc(size);
     if (!block)
-    {
-        *pExtra = 0;
         return NULL;
-    }
+#endif    
 
     assert_stack_gap();
 
@@ -3643,7 +3667,12 @@ esbrk (word_t size, size_t * pExtra)
     count_up(&sbrk_stat, size);
     count_up(&large_wasted_stat, overhead * GRANULARITY);
 
+    // amount of additional memory that was allocated
+#ifdef HAVE_MMAP
+    *pExtra = overlap + (size - overhead * GRANULARITY - req_size);
+#else
     *pExtra = overlap;
+#endif
     return block + GRANULARITY;
 #endif /* !SBRK_OK */
 } /* esbrk() */
@@ -4371,4 +4400,19 @@ mem_consolidate (Bool force)
 #endif /* MALLOC_EXT_STATISTICS */
 } /* mem_consolidate() */
 
+/*-------------------------------------------------------------------------*/
+#if !defined(HAVE_GETPAGESIZE)
+static INLINE size_t
+getpagesize()
+/* get the pagesize for this system */
+{
+#ifdef(HAVE_SYSCONF)
+    return sysconf(_SC_PAGESIZE);
+#else
+#error Unable to find out the system pagesize. Please report this issue.
+#endif
+}
+#endif /* HAVE_GETPAGESIZE */
+
 /***************************************************************************/
+
diff --git a/src/xalloc.c b/src/xalloc.c
index 89e5b58..0ae0784 100644
--- a/src/xalloc.c
+++ b/src/xalloc.c
@@ -182,7 +182,7 @@ static t_stat xalloc_stat = {0,0};
   /* Total number and size of allocations done by the driver (incl overhead).
    */
 
-#if defined(SBRK_OK)
+#if defined(MALLOC_SBRK) && (defined(SBRK_OK) || defined(HAVE_MMAP))
 static t_stat clib_alloc_stat = {0,0};
   /* Number and size of allocations done through the clib emulation
    * functions (incl overhead).
@@ -1178,7 +1178,7 @@ dump_malloc_trace (int d
 
 /*                     CLIB ALLOCATION FUNCTIONS                           */
 
-#if defined(REPLACE_MALLOC) && defined(SBRK_OK)
+#if defined(REPLACE_MALLOC) && (defined(SBRK_OK) || defined(HAVE_MMAP))
 
 /*-------------------------------------------------------------------------*/
 static POINTER
-- 
1.6.1

2009-05-26 17:16

 

0002-Use-mmap-for-getting-memory-from-the-system-small.patch (6,988 bytes)   
From c488c71a0e14e549699106dad0726e5ffc83faa5 Mon Sep 17 00:00:00 2001
From: zesstra <zesstra@zesstra.de>
Date: Thu, 21 May 2009 18:36:39 +0200
Subject: [PATCH 2/3] Use mmap() for getting memory from the system (smalloc.c)

sbrk() and brk() are obsoleted and some systems (like Darwin) don't have a
working implementation anymore. If sbrk/brk() are not usable, but mmap()
is available, use mmap() for mapping anonymous memory which we then use.
If mmap() is not available, use the old fallback (system malloc()).
Changes some precompiler checks for SBRK_OK, which should be REPLACE_MALLOC.

Signed-off-by: zesstra <zesstra@zesstra.de>
---
 src/slaballoc.c |    9 ++++---
 src/smalloc.c   |   73 +++++++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 65 insertions(+), 17 deletions(-)

diff --git a/src/slaballoc.c b/src/slaballoc.c
index e32f77e..07ad8f7 100644
--- a/src/slaballoc.c
+++ b/src/slaballoc.c
@@ -364,6 +364,7 @@
 
 /* Bitflags for the size field:
  * TODO: Assumes a word_t of at least 32 bit.
+ * TODO: define values for 64-bit word_t to support larger blocks.
  */
 #define PREV_BLOCK  0x80000000  /* Previous block is allocated */
 #define THIS_BLOCK  0x40000000  /* This block is allocated */
@@ -3463,9 +3464,10 @@ esbrk (word_t size, size_t * pExtra)
  */
 
 {
-#ifdef SBRK_OK
-
     mdb_log_sbrk(size);
+    
+#ifdef SBRK_OK
+    
     *pExtra = 0;
     if (!heap_end)
     {
@@ -3503,7 +3505,6 @@ esbrk (word_t size, size_t * pExtra)
        * the new allocation with the old one.
        */
 
-    mdb_log_sbrk(size);
     size += overhead * GRANULARITY;  /* for the extra fake "allocated" block */
 
     // get the new memory block
@@ -3622,7 +3623,7 @@ esbrk (word_t size, size_t * pExtra)
                 next = prev + *prev;
             } while (next < (word_t *)block);
             overlap = 0;
-
+                        
             if ((word_t *)block == prev + overhead)
             {
                 /* Our block directly follows the one we found */
diff --git a/src/smalloc.c b/src/smalloc.c
index f3c7c63..d9dfaed 100644
--- a/src/smalloc.c
+++ b/src/smalloc.c
@@ -213,13 +213,17 @@
 #include "driver.h"
 #include "typedefs.h"
 
-#ifdef HAVE_MADVISE
+#if defined(HAVE_MADVISE) || defined(HAVE_MMAP) || defined(HAVE_POSIX_MADVISE)
+#    include <sys/types.h>
 #    include <sys/mman.h>
 #    define MADVISE(new,old)  madvise(new,old,MADV_RANDOM)
 #else
 #    define MADVISE(new,old)  NOOP
 #endif
 
+// for sysconf()
+#include <unistd.h>
+
 #include "smalloc.h"
 
 #include "mstrings.h"
@@ -258,7 +262,11 @@
 #  else
 #      undef SBRK_OK
 #  endif
-#endif
+#else if defined (HAVE_MMAP)
+#  ifdef MALLOC_SBRK
+#      define REPLACE_MALLOC
+#  endif
+#endif // SBRK_OK
 
 #define MEM_MAIN_THREADSAFE
 
@@ -362,7 +370,7 @@
  *   CHUNK_SIZE: size of a chunk from which large blocks are allocated.
  */
 
-#ifdef SBRK_OK
+#if defined(MALLOC_SBRK) && (defined(SBRK_OK) || defined (HAVE_MMAP))
 #    define SMALL_CHUNK_SIZE    0x04000  /* 16 KByte */
 #    define CHUNK_SIZE          0x40000
 #else
@@ -379,6 +387,7 @@
 
 /* Bitflags for the size field:
  * TODO: Assumes a 32-Bit word_t.
+ * TODO: define values for 64-bit word_t to support larger blocks.
  */
 #define PREV_BLOCK  0x80000000  /* Previous block is allocated */
 #define THIS_BLOCK  0x40000000  /* This block is allocated */
@@ -777,7 +786,7 @@ mem_dump_data (strbuf_t *sbuf)
  */
 
 {
-#ifdef SBRK_OK
+#ifdef REPLACE_MALLOC
     t_stat clib_st;
 #endif
     t_stat sbrk_st, perm_st, xalloc_st;
@@ -793,7 +802,7 @@ mem_dump_data (strbuf_t *sbuf)
 #endif /* MALLOC_EXT_STATISTICS */
 
     sbrk_st = sbrk_stat;
-#ifdef SBRK_OK
+#ifdef REPLACE_MALLOC
     clib_st = clib_alloc_stat;
 #endif
     xalloc_st = xalloc_stat;
@@ -828,7 +837,7 @@ mem_dump_data (strbuf_t *sbuf)
     dump_stat("small wasted:      %8lu        %10lu (h)\n\n",s_wasted);
 
     dump_stat("permanent blocks:  %8lu        %10lu\n", perm_st);
-#ifdef SBRK_OK
+#ifdef REPLACE_MALLOC
     dump_stat("clib allocations:  %8lu        %10lu\n", clib_st);
 #else
     strbuf_addf(sbuf, "clib allocations:       n/a               n/a\n");
@@ -961,7 +970,7 @@ mem_dinfo_data (svalue_t *svp, int value)
     ST_NUMBER(DID_MEM_MINC_CALLS, malloc_increment_size_calls);
     ST_NUMBER(DID_MEM_MINC_SUCCESS, malloc_increment_size_success);
     ST_NUMBER(DID_MEM_MINC_SIZE, malloc_increment_size_total);
-#ifdef SBRK_OK
+#ifdef REPLACE_MALLOC
     ST_NUMBER(DID_MEM_CLIB, clib_alloc_stat.counter);
     ST_NUMBER(DID_MEM_CLIB_SIZE, clib_alloc_stat.size);
 #endif
@@ -3243,10 +3252,12 @@ esbrk (word_t size, size_t * pExtra)
  */
 
 {
-#ifdef SBRK_OK
-
     mdb_log_sbrk(size);
+
     *pExtra = 0;
+    
+#ifdef SBRK_OK
+
     if (!heap_end)
     {
         /* First call: allocate the first fake block */
@@ -3275,6 +3286,7 @@ esbrk (word_t size, size_t * pExtra)
 #else  /* not SBRK_OK */
 
     char *block;
+    size_t req_size = size; // the requested size of memory.
     word_t *p;    /* start of the fake block */
     const int overhead = TL_OVERHEAD;
     size_t overlap = 0;
@@ -3283,15 +3295,31 @@ esbrk (word_t size, size_t * pExtra)
        */
 
 
-    mdb_log_sbrk(size);
     size += overhead * GRANULARITY;  /* for the extra fake "allocated" block */
 
+    // get the new memory block
+#ifdef HAVE_MMAP
+    {
+        static size_t pagesize = 0; // pagesize - 1 for this system.
+        if (!pagesize)
+            pagesize = getpagesize() - 1;
+        // round to multiples of the real pagesize
+        if ((size & pagesize) != 0)
+        {
+            size += pagesize;
+            size &= ~pagesize;
+        }
+        // get new page(s)
+        block = mmap(0, size, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0);
+        if (block == MAP_FAILED)
+            return NULL;
+    }
+#else
     block = malloc(size);
     if (!block)
-    {
-        *pExtra = 0;
         return NULL;
-    }
+#endif
+    
     assert_stack_gap();
 
     /* p points to the start of the fake allocated block used
@@ -3430,7 +3458,12 @@ esbrk (word_t size, size_t * pExtra)
     count_up(&sbrk_stat, size);
     count_up(&large_wasted_stat, overhead * GRANULARITY);
 
+    // amount of additional memory that was allocated
+#ifdef HAVE_MMAP
+    *pExtra = overlap + (size - overhead * GRANULARITY - req_size);
+#else
     *pExtra = overlap;
+#endif
     return block + GRANULARITY;
 #endif /* !SBRK_OK */
 } /* esbrk() */
@@ -4017,4 +4050,18 @@ mem_consolidate (Bool force)
     } /* for (all chunks) */
 } /* mem_consolidate() */
 
+
+/*-------------------------------------------------------------------------*/
+#if !defined(HAVE_GETPAGESIZE)
+static INLINE size_t
+getpagesize()
+/* get the pagesize for this system */
+{
+#ifdef(HAVE_SYSCONF)
+    return sysconf(_SC_PAGESIZE);
+#else
+#error Unable to find out the system pagesize. Please report this issue.
+#endif
+}
+#endif /* HAVE_GETPAGESIZE */
 /***************************************************************************/
-- 
1.6.1

2009-05-26 17:16

 

0003-Removed-amalloc-afree-MALLOC_ALIGN-and-MEM_ALIGN.patch (14,690 bytes)   
From a7f48838296b7eb3351e77eab5646bcfd20df96b Mon Sep 17 00:00:00 2001
From: zesstra <zesstra@zesstra.de>
Date: Thu, 21 May 2009 20:18:52 +0200
Subject: [PATCH 3/3] Removed amalloc/afree(), MALLOC_ALIGN and MEM_ALIGN.

If we replace the system malloc, we defined amalloc() to get aligned memory.
These function used MEM_ALIGN and MALLOC_ALIGN. Unfortunately it contained
an overflow (it reserved MALLOC_ALIGN-MEM_ALIGN, but wrote up to
MALLOC_ALIGN chars).
Additionally, it never worked. If using slaballoc/smalloc/sysmalloc,
MEM_ALIGN and MALLOC_ALIGN were always equal and in case of xptmalloc
MALLOC_ALIGN was always <= MEM_ALIGN. Therefore, amalloc never added
anything for alignment.
Furthermore, if malloc/amalloc() should use the same alignment as the system
allocator, our amalloc() was not sufficient.
We may re-introduce allocator functions for aligned memory if needed (the
driver does not), but then we should do it properly (and ensure to have the
same alignment as the system allocator).

Signed-off-by: zesstra <zesstra@zesstra.de>
---
 src/autoconf/configure.in |   10 ----
 src/slaballoc.c           |   42 +++++++++------
 src/smalloc.c             |   46 ++++++++++-------
 src/sysmalloc.c           |    1 -
 src/xalloc.c              |  124 ++++----------------------------------------
 src/xptmalloc.c           |    4 --
 6 files changed, 65 insertions(+), 162 deletions(-)

diff --git a/src/autoconf/configure.in b/src/autoconf/configure.in
index 0077d9a..cdd5540 100644
--- a/src/autoconf/configure.in
+++ b/src/autoconf/configure.in
@@ -817,16 +817,6 @@ AC_CHECK_FUNCS(fchmod getrusage bzero memset memcpy memmem strdup strcspn)
 AC_CHECK_FUNCS(strchr strrchr getcwd memmove sysconf gettimeofday wait3 waitpid)
 AC_CHECK_FUNCS(fcntl getdomainname poll strtoul trunc)
 
-
-AC_CACHE_CHECK(for needed malloc() alignment,lp_cv_sys_malloc_align,
-AC_TRY_COMPILE([struct ts {double d; char *p; double e;};
-int i = 96/(sizeof(struct ts)-20);
-],,
-lp_cv_sys_malloc_align=8,
-lp_cv_sys_malloc_align=4))
-AC_DEFINE_UNQUOTED(MALLOC_ALIGN,$lp_cv_sys_malloc_align,[word alignment])
-
-
 if test "x$ac_cv_type_signal" = "xvoid"; then
   AC_DEFINE(RETSIGTYPE_VOID, 1,                                                                                                                                                                                    
     [Set in response to the signal handler return type, since not all                                                                                                                                              
diff --git a/src/slaballoc.c b/src/slaballoc.c
index 07ad8f7..2725aef 100644
--- a/src/slaballoc.c
+++ b/src/slaballoc.c
@@ -110,12 +110,25 @@
  * A new AVL node is created for every free large block.
 #endif
  *
-#ifdef SBRK_OK && MALLOC_SBRK
+#ifdef MALLOC_SBRK && SBRK_OK
  * Memory is allocated from the system with sbrk() and brk(), which puts
  * the whole heap under our control.
  *
- * In this mode, several functions like amalloc() are compiled as malloc(),
- * implementing all libc memory functions.
+ * In this mode, we replace the system malloc() and free() by our own 
+ * functions, implementing all libc memory functions (malloc, free, calloc,
+ * realloc()).
+#else if MALLOC_SBRK && HAVE_MMAP
+ * If brk/sbrk() are non-functional (e.g. on Darwin; remember: they are legacy
+ * and not POSIX anymore), we check if mmap() is available. If it is, we use 
+ * it to map anonymous memory pages to get memory from the VM system.
+ *
+ * The allocated block (modulo joints) is tagged at both ends with fake
+ * "allocated" blocks of which cover the unallocated areas - large_malloc()
+ * will perceive this as a fragmented heap.
+ *
+ * In this mode, we replace the system malloc() and free() by our own 
+ * functions, implementing all libc memory functions (malloc, free, calloc,
+ * realloc()). 
 #else
  * malloc() is used to allocate a new block of memory. If this block borders
  * previous ones, the blocks are joined.
@@ -232,25 +245,19 @@
  */
 #define GRANULARITY sizeof(word_t)
 
-/* Defines required by the xalloc.c wrapper */
-#define MEM_ALIGN GRANULARITY
-
 /* If possible, request memory using sbrk(). But if using pthreads, do
  * not replace malloc() with our routines, even if the system allows it,
  * as slaballoc is not threadsafe.
+ * If sbrk/brk() are not working, but mmap() is, then use mmap() for getting
+ * memory. If that is also not available, use malloc().
  */
-
 #if defined(SBRK_OK)
 #  ifdef MALLOC_SBRK
-#      if MALLOC_ALIGN == 4
-#          define REPLACE_MALLOC
-#      else
-#          undef SBRK_OK
-#      endif
+#      define REPLACE_MALLOC
 #  else
 #      undef SBRK_OK
 #  endif
-#else if defined (HAVE_MMAP)
+#else if defined(HAVE_MMAP)
 #  ifdef MALLOC_SBRK
 #      define REPLACE_MALLOC
 #  endif
@@ -605,12 +612,15 @@ static unsigned long num_avl_nodes = 0;
 /*-------------------------------------------------------------------------*/
 /* Forward declarations */
 
-static char *esbrk(word_t, size_t * pExtra);
+static char *esbrk(word_t size, size_t * pExtra) __attribute__((malloc,warn_unused_result));
 
-static char *large_malloc(word_t, Bool);
+static char *large_malloc(word_t size, Bool force_m) __attribute__((malloc,warn_unused_result));
 #define large_malloc_int(size, force_m) large_malloc(size, force_m)
 static void large_free(char *);
 
+static INLINE size_t mem_overhead (void) __attribute__((const));
+
+
 #ifdef MALLOC_EXT_STATISTICS
 /*=========================================================================*/
 
@@ -726,7 +736,7 @@ mem_block_size (POINTER p)
 
     if (q[M_SIZE] & M_SMALL)
     {
-        return mem_block_total_size(p) - M_OVERHEAD*GRANULARITY;
+        return mem_block_total_size(p) - mem_overhead();
     }
     return mem_block_total_size(p) - ML_OVERHEAD*GRANULARITY;
 } /* mem_block_size() */
diff --git a/src/smalloc.c b/src/smalloc.c
index d9dfaed..e657e7c 100644
--- a/src/smalloc.c
+++ b/src/smalloc.c
@@ -112,20 +112,33 @@
  * A new AVL node is created for every free large block.
 #endif
  *
-#ifdef SBRK_OK && MALLOC_SBRK
+ #ifdef MALLOC_SBRK && SBRK_OK
  * Memory is allocated from the system with sbrk() and brk(), which puts
  * the whole heap under our control.
  *
- * In this mode, several functions like amalloc() are compiled as malloc(),
- * implementing all libc memory functions.
-#else
+ * In this mode, we replace the system malloc() and free() by our own 
+ * functions, implementing all libc memory functions (malloc, free, calloc,
+ * realloc()).
+ #else if MALLOC_SBRK && HAVE_MMAP
+ * If brk/sbrk() are non-functional (e.g. on Darwin; remember: they are legacy
+ * and not POSIX anymore), we check if mmap() is available. If it is, we use 
+ * it to map anonymous memory pages to get memory from the VM system.
+ *
+ * The allocated block (modulo joints) is tagged at both ends with fake
+ * "allocated" blocks of which cover the unallocated areas - large_malloc()
+ * will perceive this as a fragmented heap.
+ *
+ * In this mode, we replace the system malloc() and free() by our own 
+ * functions, implementing all libc memory functions (malloc, free, calloc,
+ * realloc()). 
+ #else
  * malloc() is used to allocate a new block of memory. If this block borders
  * previous ones, the blocks are joined.
  *
  * The allocated block (modulo joints) is tagged at both ends with fake
  * "allocated" blocks of which cover the unallocated areas - large_malloc()
  * will perceive this as a fragmented heap.
-#endif
+ #endif
  *
  *  -- Privilege Level --
  *
@@ -245,20 +258,13 @@
  */
 #define GRANULARITY sizeof(word_t)
 
-/* Defines required by the xalloc.c wrapper */
-
-#define MEM_ALIGN GRANULARITY
-
 /* If possible, request memory using sbrk().
+ * If sbrk/brk() are not working, but mmap() is, then use mmap() for getting
+ * memory. If that is also not available, use malloc().
  */
-
 #if defined(SBRK_OK)
 #  ifdef MALLOC_SBRK
-#      if MALLOC_ALIGN == 4
-#          define REPLACE_MALLOC
-#      else
-#          undef SBRK_OK
-#      endif
+#      define REPLACE_MALLOC
 #  else
 #      undef SBRK_OK
 #  endif
@@ -603,11 +609,13 @@ static long num_avl_nodes = 0;
 /*-------------------------------------------------------------------------*/
 /* Forward declarations */
 
-static char *esbrk(word_t, size_t * pExtra);
+static char *esbrk(word_t size, size_t * pExtra) __attribute__((malloc,warn_unused_result));
 
-static char *large_malloc(word_t, Bool);
+static char *large_malloc(word_t size, Bool force_m) __attribute__((malloc,warn_unused_result));
 #define large_malloc_int(size, force_m) large_malloc(size, force_m)
-static void large_free(char *);
+static void large_free(char *ptr);
+static INLINE size_t mem_overhead(void) __attribute__((const));
+
 
 #ifdef MALLOC_EXT_STATISTICS
 /*=========================================================================*/
@@ -695,7 +703,7 @@ mem_block_size (POINTER p)
     word_t size = (q[M_SIZE] & M_MASK);
     if (size > SMALL_BLOCK_MAX)
         return mem_block_total_size(p) - ML_OVERHEAD*GRANULARITY;
-    return mem_block_total_size(p) - M_OVERHEAD*GRANULARITY;
+    return mem_block_total_size(p) - mem_overhead();
 } /* mem_block_size() */
 
 /*-------------------------------------------------------------------------*/
diff --git a/src/sysmalloc.c b/src/sysmalloc.c
index 5e0b81a..49c59e1 100644
--- a/src/sysmalloc.c
+++ b/src/sysmalloc.c
@@ -17,7 +17,6 @@
 #include "../mudlib/sys/debug_info.h"
 
 /* Defines required by the xalloc.c wrapper */
-#define MEM_ALIGN (MALLOC_ALIGN)
 /* #undef REPLACE_MALLOC */
 #define NO_MEM_BLOCK_SIZE
 #define MEM_THREADSAFE
diff --git a/src/xalloc.c b/src/xalloc.c
index 0ae0784..a5c6b2c 100644
--- a/src/xalloc.c
+++ b/src/xalloc.c
@@ -113,8 +113,6 @@ enum xalloc_header_fields {
     XM_OVERHEAD_SIZE = XM_OVERHEAD * sizeof(word_t),
 };
 
-#define XM_OVERHEAD_SIZE (XM_OVERHEAD * sizeof(word_t))
-
 /*-------------------------------------------------------------------------*/
 
 /* -- Global Variables/Arguments dealing with memory -- */
@@ -323,8 +321,6 @@ mdb_log_sbrk (p_int size)
 #endif
 #endif
  *
- *   #define MEM_ALIGN
- *     the alignment guaranteed by the allocator
  *   #define REPLACE_MALLOC
  *     if the allocator's mem_alloc()/mem_free() can be used to replace the
  *     libc allocation routines (ie. the allocator doesn't use malloc()
@@ -1181,122 +1177,26 @@ dump_malloc_trace (int d
 #if defined(REPLACE_MALLOC) && (defined(SBRK_OK) || defined(HAVE_MMAP))
 
 /*-------------------------------------------------------------------------*/
-static POINTER
-amalloc (size_t size)
+POINTER
+malloc (size_t size)
 
-/* Allocate an aligned block of <size> bytes, if necessary, with
- * SYSTEM privilege. The block is not subject to GC.
- * Result is the pointer to the allocated block, or NULL.
+/* Allocate an empty memory block of size <sizel>.
+ * The memory is aligned and not subject to GC.
  */
 
 {
-    char *temp;
-    size_t orig_size UNUSED; // unused if no HAVE_MADVISE.
-    
-    if (MALLOC_ALIGN > MEM_ALIGN)
-    {
-        
-#if defined(HAVE_MADVISE)
-        orig_size = size;
-#endif
-        
-        size += (MALLOC_ALIGN-MEM_ALIGN);
-    }
-
-    temp = (char *)pxalloc(size);
-    if (!temp)
+    POINTER result = pxalloc(size);
+    if (!result)
     {
         int save_privilege = malloc_privilege;
         malloc_privilege = MALLOC_SYSTEM;
-        temp = (char *)pxalloc(size);
+        result = pxalloc(size);
         malloc_privilege = save_privilege;
     }
-
-    if (MALLOC_ALIGN > MEM_ALIGN) 
-    {
-        if (temp)
-        {
-            /* Set the first byte of the alignment area to 0xAF - afree(0
-             * is going to look for it - and the rest to 0.
-             */
-            *temp++ = 0xAF;
-            while ((p_int)temp & (MALLOC_ALIGN-1))
-                *temp++ = 0;
-            MADVISE(temp, orig_size);
-        }
-    }
-
-    return (POINTER)temp;
-} /* amalloc() */
-
-/*-------------------------------------------------------------------------*/
-static void
-afree (POINTER p)
-
-/* Free the aligned memory block <p>.
- */
-
-{
-    char *q = (char *)p;
-
-    if (!q)
-        return;
-
-    if (MALLOC_ALIGN > MEM_ALIGN)
-    {
-        
-        /* amalloc() filled the alignment area with 0s except for the first byte.
-         * Search backwards to find that marker and with it the real block begin.
-         */
-        while (!*--q) NOOP;
-    }
-
-    pfree(q);
-} /* afree() */
-
-/*-------------------------------------------------------------------------*/
-static INLINE long
-get_block_size (POINTER ptr)
-
-/* Get the allocated block size for the block with user area starting
- * at <ptr>. This function is meant only for block allocated with (a)malloc().
- * Result is the size in bytes inclusive overhead.
- */
-{
-    long size = 0;
-
-    if (ptr)
-    {
-        /* Get the allocated size of the block for the statistics */
-
-        char *q;
-
-        q = (char *)ptr;
-        
-        if (MALLOC_ALIGN > MEM_ALIGN)
-            while ( !(size = *--q) ) NOOP;
-
-        size = xalloced_size(q) + mem_overhead();
-    }
-
-    return size;
-} /* get_block_size() */
-
-/*-------------------------------------------------------------------------*/
-POINTER
-malloc (size_t size)
-
-/* Allocate an empty memory block of size <sizel>.
- * The memory is aligned and not subject to GC.
- */
-
-{
-    POINTER result;
-
-    result = amalloc(size);
+    
     if (result)
     {
-        count_up(&clib_alloc_stat, get_block_size(result));
+        count_up(&clib_alloc_stat, xalloced_size(result) + mem_overhead());
     }
 
     return result;
@@ -1312,10 +1212,10 @@ free (POINTER ptr)
 {
     if (ptr)
     {
-        count_back(&clib_alloc_stat, get_block_size(ptr));
+        count_back(&clib_alloc_stat, xalloced_size(ptr) + mem_overhead());
     }
 
-    afree(ptr);
+    pfree(ptr);
     FREE_RETURN
 } /* free() */
 
@@ -1354,7 +1254,7 @@ realloc (POINTER p, size_t size)
    if (!p)
         return malloc(size);
 
-   old_size = get_block_size(p) - xalloc_overhead();
+   old_size = xalloced_size(p) - xalloc_overhead();
 
    if (old_size >= size)
       return p;
diff --git a/src/xptmalloc.c b/src/xptmalloc.c
index d8fa66f..d467b11 100644
--- a/src/xptmalloc.c
+++ b/src/xptmalloc.c
@@ -232,10 +232,6 @@ static Bool mem_is_freed (POINTER p, size_t minsize) {
 
 #endif /* GC_SUPPORT */
 
-/*     the alignment guaranteed by the allocator */
-//#define MEM_ALIGN (2*(sizeof(size_t)))
-#define MEM_ALIGN (2*SIZEOF_INT)
-
 /* If the allocator can replace the libc allocation routines.
  * See above for the *BSD situation.
  */
-- 
1.6.1

zesstra

2009-05-26 17:20

administrator   ~0001165

I attached 3 patches for adding mmap support to slaballoc.c and smalloc.c
The third patch removes amalloc() and MALLOC_ALIGN. The reasoning for this is:

If we replace the system malloc, we defined amalloc() to get aligned memory.
These function used MEM_ALIGN and MALLOC_ALIGN. Unfortunately it contained
an overflow (it reserved MALLOC_ALIGN-MEM_ALIGN, but wrote up to
MALLOC_ALIGN chars).
Additionally, it never worked. If using slaballoc/smalloc/sysmalloc,
MEM_ALIGN and MALLOC_ALIGN were always equal and in case of xptmalloc
MALLOC_ALIGN was always <= MEM_ALIGN. Therefore, amalloc never added
anything for alignment.
Furthermore, if malloc/amalloc() should use the same alignment as the system
allocator, our amalloc() was not sufficient.
We may re-introduce allocator functions for aligned (other than the word_t which the allocators guarantee) memory if needed (the driver does obviously not), but then we should do it properly (and ensure to have the same alignment as the system allocator).

zesstra

2009-05-28 16:16

administrator   ~0001173

This should be resolved by r2604 - r2606.

Issue History

Date Modified Username Field Change
2009-05-22 06:55 zesstra New Issue
2009-05-22 06:55 zesstra Status new => assigned
2009-05-22 06:55 zesstra Assigned To => zesstra
2009-05-26 17:16 zesstra File Added: 0001-Use-mmap-for-getting-memory-from-the-system-slaba.patch
2009-05-26 17:16 zesstra File Added: 0002-Use-mmap-for-getting-memory-from-the-system-small.patch
2009-05-26 17:16 zesstra File Added: 0003-Removed-amalloc-afree-MALLOC_ALIGN-and-MEM_ALIGN.patch
2009-05-26 17:20 zesstra Note Added: 0001165
2009-05-28 16:16 zesstra Note Added: 0001173
2009-05-28 16:16 zesstra Status assigned => resolved
2009-05-28 16:16 zesstra Fixed in Version => 3.3.719
2009-05-28 16:16 zesstra Resolution open => fixed