How to do Binary instrumentation of syscall brk ? (x86-64 Linux) (maybe valgrind?) - valgrind

Question

I'd like to instrument syscall brk (and other calls but this in first order, it's most important to me) in given binary (preferably on actual syscall/sysenter level (x86-64 and x86) of making sys_brk call).
Main goal:
A part of sandbox which gives fixed amount of memory to jailed process
So, I'd like to get rid of brk system calls (and most preferably others in next order) and simulate memory allocations under fixed limit. Fixed limit is memory space, available to program. (You can think about it like making a kind of sandbox with fixed amount of available memory)
How to implement (one of) some example possible solutions (or yours solution):
just changing instructions to NOP
As brk returns 0 on success, simulate it's successes with setting operations that sets memory (register) state , as brk would be called with success.
More complex... instrument with code (or function call) which simulates success memory allocations under fixed limit.
Most flexible (maybe overkill in my case) to change this syscall into function call and add provided function to binary.
Given binary is code that can be malicious in one of two (most preferably both :) ) forms:
shared library - here I can setup environment before function call (for example do brk call in controlled way)
program binary - in this case we need to give program fixed amount of memory (by caller, or on begining of program "one syscall"), cause it can not allocate. Example of calling such program should be included in answer.
As problem is highly connected with many other aspects, I tried do my best in separating it as question, but please give me advice if I should specify something more or less.
Answers with implementation, links to resources (books, tutorials) are welcome.
(I am most interested in Linux, and solution that is reliable, so that people preparing binaries, even in assembler, would not have to worry about execution of their code)

Answer 1

LD_PRELOAD will trap C calls to brk(), but it won't trap the actual system call (int/syscall instruction). There's no portable way to trap those, but on Linux, ptrace will do it. Memory can also be allocated to a program by mmap(), so you'll need to intercept that call too.
Of course, what it seems you're really looking for is rlimit().

Answer 2

Yeah, I don't think you want valgrind for this.
You can use LD_PRELOAD or linker tricks to capture brk(2): see these other discussions:
Function interposition in Linux without dlsym
Overriding 'malloc' using the LD_PRELOAD mechanism
Code might look like this:
#include <unistd.h>
#include <dlfcn.h>
/* prototype int brk(void *addr); */
static int (*real_brk)(void *addr) = NULL;
int brk(void * addr) {
real_brk = dlsym(RTLD_NEXT, "brk");
if (real_brk == NULL) {
fprintf(stderr, "error mapping brk: %s\n", dlerror());
return -1;
}
printf("calling brk(2) for %p\n", addr);
return (real_brk (addr));
}`
and then LD_PRELOAD that to intercept brk(2)