I think of stack and heap as referring to different sections of a process' virtual memory, where one grows upwards and the other downwards. Physical memory addresses (in the form of pages) are mapped to these virtual addresses. AFAIK, the addressing of the virtual address space is from 0x00000000 to 0xFFFFFFFF, which is humongous. Since the area between the stack and heap is so huge (most likely much larger than the actual physical memory on the computer), I don’t think the stack and heap can ever meet each other. Also, what then limits the amount of heap memory a program can ask from the OS?
Related
I've been studying about paging and page tables. I don't see to understand where page tables are located.
In one of the answers from stack exchange(https://unix.stackexchange.com/questions/487052/where-is-page-table-stored-in-linux), it is said that page tables are in kernel address space, which is in virtual memory(from what I understood).
However in lecture slides from University of Illinois(https://courses.engr.illinois.edu/cs241/sp2014/lecture/09-VirtualMemory_II_sol.pdf), page tables seem to be in RAM, which is physical memory.
Can anyone tell me clearly where the page tables are stored?
Thank you in advance.
The answer to this question is too broad, and I think it belongs to super-user stack exchange.
In x86 systems, page tables are structures used by the CPU, but they are too large to be hold in registers, so they are kept in RAM.
Any process has a memory map in which there is two big zones: user space and kernel space. Kernel space is the same space for all process. User space is private to that process. On 32 bit X86 based Linux systems, any logical address equal or greater than 0xC0000000 belongs to kernel. Below that address, it's user space.
The page table of the process is held in the kernel space. The kernel may have several page tables in RAM, but only one is the active page table. In x86 CPUs, it's the page table pointed by register CR3.
There is a more detailed explanation of how it works here: https://stackoverflow.com/a/20792205/3011009
i think you have a problem about understanding the virtual and physical memory.
as the name suggest the virtual memory is not real. the reason of the idea of virtual memory was that the process sees all the storage in a computer as the available memory. for example in a 64 bit system, a process might see 2^64 as the memory available to it and another process may see the same thing. so using the virtual memory every process would see a continuous memory available to it which might be so much bigger than the available memory on the system. all the addresses in the virtual memory then should be translated to the equivalent physical memory using something called page tables.
pages are blocks of cells(addresses), for example lets say that the available memory(physical) in a system is 2 GB, and the pages or blocks of cells has been chosen as 4 KB, in this case in a 4 KB block or page 4096 different cells or addresses are available which we could address using 12 bits , since we have:
2^12 = 4096
if the overall memory is 2 GB, then it means we could have:
2GB/4KB = 524288
which means we could have 524288 different pages in the physical memory, now some of these pages are only assigned to the operating system code, which means only the os could have access to it, these are the codes and instructions of the operating system program which could help the execution of every other program. other pages are available for other processes.
now lets say we have an address like this in the virtual memory:
0x000075fe
first of all we said that we need 12 bits to tell the position of every address in the page itself since the page is 4 KB, this position is 5fe, what operating system or every other memory management tool does! is that it won't translate this OFFSET, the position of every address in the virtual page would be the same thing in the physical page, i think this is one of the main features which makes translation beneficial , now the rest of the address should be translated to the related page in the physical which is :
0x00007
for this , the page table should be looked, which as we said is just a table in the kernel memory, which is not accessible in the user space, for example is something like this:
0x00001 0x00004
0x00002 disk ----> means every these addresses are in the disk
0x00007 0x004fe
so the 0x00007 page should be translated to the 0x004fe and therefore the address of:
0x000075fe in the virtual memory would be translated to:
0x004fe5fe in the physical memory , which means this is an address in the page number 0x004fe and the position of 5feth - 1.(since we know the starting point is zero).
Observation is-
2 threads a and b are created one after the other. Stack size required for thread b is thrice of the allocated stack size.
Execution of thread b uses/corrupts stack allocated to thread a. Now when thread a executes, OS gives stack overflow error.
Note: New functions are added to thread b. No new function executes in the context of thread a.
RTOS used is embOS.
Is this scenario possible? Or the interpretation of the observations might be wrong in this case? How can I find out?
The scenario is very likely, somewhat depending on the layout of your memory; but almost certainly something bad will happen. embOS normally does no memory protection, since most systems it runs on don't have an MMU to protect you memory.
On most architectures stacks grow downwards, i.e. from the upper memory addresses to the lower ones. If you create your stacks like the following (common to do so when using embOS):
static char stack_a[512];
static char stack_b[512];
and thread b uses 1536 bytes of stack memory, it will use stack_b for its first 512 bytes, stack_a for the next 512 bytes and 512 bytes from something we can't see here. So bad things will happen...
The reason that embOS detects your stack corruption is (with the configurations I know), that it put some special signature to the lower bytes of the stack and checks at some occasions if this signature is still intact. This although implies, that you need more than 512 bytes of stack space when you want to use 512 byte stack memory in your thread. Having some reserve on your stacks is almost always a good idea.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
I am running Ubuntu on my physical machine; and VirtualBox to run various virtual ones on top.
Most of the time, I was doing "straight forward" installations; but today I wanted to be "smart" and checked out the partitions that the fedora or ubuntu installers will create on my virtual disks.
And sure, when going for the defaults, some GBs of my virtual disks will be used as "swap space".
Now I am wondering: assuming that I have plenty of physical memory (so I can assign 6 or 8 GB to a VM) - is there any sense in dedicated swap space for a a virtual machine?
This is answered at ServerFault:
TL;DR: use swap: 1. avoids out of memory error, 2. guest OS is better at memory management
Ignoring the fact that people are dealing with OS specific reasons I
have two reasons why it's a bad idea to not run with a swap
partition/file.
If you have 1.5 GB of RAM allocated to a VM with no space file/partition and it wants to use 1.5 GB + 1 MB it will report an out
of memory error. With the swap space it will be able to swap data out
of active memory and onto the disk.
The guest OS does a much better job of memory management than the host. This is why technology like memory ballooning exists because the
Host can make educated guesses on what memory isn't needed right now
but the guest knows at a much more intelligent level (this keeps OS
memory from being swapped out which could kill your performance).
Swap partitions are used to free your physical memory when it goes out of space. In modern day machines, with plenty of memory, it depends on the type of applications you would be running. If you are planning to run such memory intensive programs like video editors, high end games or something of that sort, virtual memory or swap space is an asset. But if it is not the case then you are safe to avoid swap space, provided you have enough memory. But it is safe to have a fallback.
That depends on what programs you are running on your host system along with the virtual machine, or what programs you are running within the virtual machine.
The only upper bound on memory that software can consume is the total memory (physical or virtual) available to it. There are plenty of programs that require large amounts of memory when behaving normally, and plenty of circumstances that cause a program to consume large amounts of memory (e.g. loading of input files). There are also plenty of faulty programs that unintentionally consume large amounts of memory.
You can often get an idea by examining requirements or recommendations (e.g. memory and drive space) of the programs you run. Failing that, try it out.
I've encountered the problem that accessing data stored in heap memory performs really slow when the memory is frequently reallocated.
in comparison to
What could explain this behaviour?
Possibly page fault issues. If you malloc a large block of RAM, the physical RAM will probably not be allocated straight away, some page table entries will be set. The physical RAM won't be allocated until you access a location in it for the first time. This involves
a page fault,
finding a physical memory page
zeroing every location on that page
updating the page table
This is an expensive operation in terms of time and will happen once per allocated page (550 x 4kbyte pages for the RAM you are allocating)
In a typical C program, the linux kernel provides 84K - ~100K of memory. How does the kernel allocate more memory for the stack when the process uses the given memory.
IMO when the process takes up all the memory of the stack and now uses the next contiguous memory, ideally it should page fault and then the kernel handles the page fault.
Is it here that the kernel provides more memory to the stack for the given process, and which data structure in linux kernel identifies the size of the stack for the process??
There are a number of different methods used, depending on the OS (linux realtime vs. normal) and the language runtime system underneath:
1) dynamic, by page fault
typically preallocate a few real pages to higher addresses and assign the initial sp to that. The stack grows downward, the heap grows upward. If a page fault happens somewhat below the stack bottom, the missing intermediate pages are allocated and mapped. Effectively increasing the stack from the top towards the bottom automatically. There is typically a maximum up to which such automatic allocation is performed, which can or can not be specified in the environment (ulimit), exe-header, or dynamically adjusted by the program via a system call (rlimit). Especially this adjustability varies heavily between different OSes. There is also typically a limit to "how far away" from the stack bottom a page fault is considered to be ok and an automatic grow to happen. Notice that not all systems' stack grows downward: under HPUX it (used?) to grow upward so I am not sure what a linux on the PA-Risc does (can someone comment on this).
2) fixed size
other OSes (and especially in embedded and mobile environments) either have fixed sizes by definition, or specified in the exe header, or specified when a program/thread is created. Especially in embedded real time controllers, this is often a configuration parameter, and individual control tasks get fix stacks (to avoid runaway threads taking the memory of higher prio control tasks). Of course also in this case, the memory might be allocated only virtually, untill really needed.
3) pagewise, spaghetti and similar
such mechanisms tend to be forgotten, but are still in use in some run time systems (I know of Lisp/Scheme and Smalltalk systems). These allocate and increase the stack dynamically as-required. However, not as a single contigious segment, but instead as a linked chain of multi-page chunks. It requires different function entry/exit code to be generated by the compiler(s), in order to handle segment boundaries. Therefore such schemes are typically implemented by a language support system and not the OS itself (used to be earlier times - sigh). The reason is that when you have many (say 1000s of) threads in an interactive environment, preallocating say 1Mb would simply fill your virtual address space and you could not support a system where the thread needs of an individual thread is unknown before (which is typically the case in a dynamic environment, where the use might enter eval-code into a separate workspace). So dynamic allocation as in scheme 1 above is not possible, because there are would be other threads with their own stacks in the way. The stack is made up of smaller segments (say 8-64k) which are allocated and deallocated from a pool and linked into a chain of stack segments. Such a scheme may also be requried for high performance support of things like continuations, coroutines etc.
Modern unixes/linuxes and (I guess, but not 100% certain) windows use scheme 1) for the main thread of your exe, and 2) for additional (p-)threads, which need a fix stack size given by the thread creator initially. Most embedded systems and controllers use fixed (but configurable) preallocation (even physically preallocated in many cases).
edit: typo
The stack for a given process has a limited, fixed size. The reason you can't add more memory as you (theoretically) describe is because the stack must be contiguous, and it grows toward the heap. So, when the stack reaches the heap, no extension is possible.
The stack size for a userland program is not determined by the kernel. The kernel stack size is a configuration option for the kernel (usually 4k or 8k).
Edit: if you already know this, and were merely talking about the allocation of physical pages for a process, then you have the procedure down already. But there's no need to keep track of the "stack size" like this: the virtual pages in the stack with no pagetable entries are just normal overcommitted virtual pages. Physical memory will be granted on their first access. But the kernel does not have to overcommit memory, and thus a stack will probably have complete physical realization when the executable is first loaded.
The stack can only be used up to a certain length, because it has a fixed storage capacity in memory. If your question asks in what direction does the stack being used up? the answer is downwards. It is filled down in memory towards the heap. The heap is a dynamic component of memory by which it can actually grow from the bottom up, based on your need of data storage.