Memory Allocations and Garbage Collection

[mahrud]

I'm curious about the nuances of memory allocation (malloc) in Macaulay2, at the engine, interpreter, and top level, and I'm hoping to use this space to share insights, best practices, etc.

I'll start off with some questions.

I. Engine

1. what's the difference between different malloc libraries? I know of:
   - malloc from GNU C
   - jemalloc used in Rust
   - tcmalloc
   - malloc substitute from bdw-gc
   - some more
2. are there examples of other programs that have something like our_new_delete?
3. is it possible to hot-swap the malloc library that engine uses with a different one for comparison? (relevant: replacing malloc in C and in Rust)
4. is it possible to hot-swap the malloc library that bdw-gc uses?

II. Interpreter

1. has there been an audit of the interpreter's memory usage?
2. can we know whether the gc is doing easily avoidable work collecting stuff that the interpreter leaves around?
3. can we know which functions run without any dynamic allocations and which don't?
4. (probably more to add later)

III. Top level

1. is it possible to detect memory allocation at the top level?
2. what are the malloc calls associated to calling a function?
3. where does tail-end optimization happen? are there other similar optimizations?
4. what's causing the issue in memory leak #488?

[mahrud]

Addendum: some interesting references about garbage collectors:

• High-performance GC for C++
• Oilpan
• Cross-Component GC
• V8 GC
• memory leaks
• Julia GC

[mahrud]

An interesting feature of Julia's GC:

julia> @time G = GroebnerBasis.f4(I);
  0.523836 seconds (676.75 k allocations: 39.873 MiB, 7.45% gc time, 77.35% compilation time)

[jkyang92]

Here's some cool stuff with heap profiling that GHC (haskell) uses https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/profiling.html#profiling-memory-usage. In particular it can plot memory usage by type or by source as a function of time.