what is the exact difference between process control block and process descriptor?.
I was reading about kernel of linux. It was written that there is some thread_info structure which contains the pointer to actual process descriptor table. It was written that the thread_info lies just above/below of kernel stack. So definitely thread_info is in main memory. But what about actual process descriptor task_struct? where is it located? If process descriptor resides in main memory, where is the actual place for it ?
The thread_info and task_struct structures are just two different structures that hold different pieces of information about a thread, with the thread_info holding more architecture-specific data than the task_struct. It makes more sense to split up the information rather than keep it all in the same structure. (Although you could put them in the same struct; the 2.4 Linux kernel did this.)
How those structs are allocated depends on the architecture you're using. The relevant functions you want to examine are alloc_task_struct() and alloc_thread_info().
In the kernel, the process descriptor is a structure called task_struct, which keeps track of process attributes and information. All kernel information regarding a process is found there.
Related
If we talk about Address Space of a process it is the virtual address range which includes static data, stack and heap memory for that particular process. And coming to Process Control Block (PCB) which is a data structure maintained by operating system for each process it manages, where PCB includes a lot of information about the process like process no., process state, program counter, list of open files, cpu scheduling info ...and more.
Now this is the point where I got confused that Address Space is also a memory which stores information about a process and similar thing is done by PCB too. Then how these are connected to each other. I am not able to visualize this in my mind. Why we have these two things existing simultaneously. Isn't it possible to achieve our goal just by using PCB?
Process Address space refer to memory regions the process is using. It typically consists of heap, stack, initialized data, uninitialized data and text. There are mainly two address spaces of a process -- logical and physical.
The PCB is a structure resides in kernel to track the state of process. One of the things PCB contain is memory information. In a typically system, PCB may contain information about pages the process has.
To answer your question, Process Address space is an idea built on top of PCB and many other things (such as page table).
This seems to be a simple Vulkan API question but I really can not find answer after search Internet.
I noticed there is a Vulkan function:
void vkCmdUpdateBuffer(
VkCommandBuffer commandBuffer,
VkBuffer dstBuffer,
VkDeviceSize dstOffset,
VkDeviceSize dataSize,
const void* pData);
At first glance, I thoughts it can be used to record the command buffer since it has prefix vkCmd in its name, but the document says that
vkCmdUpdateBuffer is only allowed outside of a render pass. This command is treated as “transfer” operation, for the purposes of synchronization barriers.
So I start thinking that it is a convenience function that wraps the buffer data transferring operation like using memcpy() to copy the data from host to the device.
Then my question is: Why there is NOT a single Vulkan sample / tutorial (I have searched all of them) using vkCmdUpdateBuffer() instead of manually coping data by memcpy(). Did I understand it wrong?
All vkCmd* functions generate commands into a command buffer. This one is no exception. It is a transfer command, and like most transfer commands, you don't get to do them within a render pass. But there are plenty of command buffer generating commands that don't work in render passes.
Normally Vulkan memory transfer operations only happen between device memory. The typical mechanism for the host to put something in device memory is to write to a mapped pointer. But by definition, that requires that the destination memory be mappable. So if you want to write something to non-mappable memory, you have to copy it to mappable memory, then do a transfer operation between the mappable memory to the non-mappable memory via vkCmdCopy* functions.
And that's fine if you're doing a bunch of transfers all at once. You can copy a bunch of stuff into mapped memory, then submit a batch containing all of the copy operations to copy the data into the appropriate locations.
But sometimes, you're just updating a small piece of device memory. If it's not mappable, then that's a lot of work to do just to get a few kilobytes of data to the GPU. In that case, vkCmdUpdateBuffer may be the better choice, since it can "directly" copy from CPU memory to any device memory.
I say "directly" because that's obviously not what it's doing. It's really doing the same thing you would have done, except it's doing it within the command buffer. You would have copied your CPU data into GPU mappable memory, then created a command that copies from that mappable memory into non-mappable memory.
vkCmdUpdateBuffer does the exact same thing. It copies the data from the pointer/size you give it into mappable memory (which is provided by the command buffer itself. This is why it has an upper limit of 64KB). This copy happens immediately, just as it would have if you did a memcpy, so when this function returns, you can do whatever you want with the pointer you gave it. Then it creates a command in the command buffer that copies from the mappable memory in the command buffer to the destination memory location.
The documentation for this function explicitly gives warnings about using it for larger transfers. That is, it tells you not to do that. This is for quick, small, one-shot updates of unmappable memory. Nothing more.
That's one reason why tutorials don't talk about it: it's a highly special-case function that many novice users will try to use because it's easier than the explicit code. But in most cases, they should not be using it.
I've watched the presentation and still have one question about working of shared buffers. As the slide 16 shows, when the server handles an incoming request, the postmaster process calls fork() to create a child one for handling the incoming request. Here is a picture from there:
So, we have the entire copy of the postmaster process except its pid. Now, if the child process update some data belonging to shared memory (putting in shared buffers, as shown in the slide 17), we need the other threads be awared of the changes. The picture:
The synchronization process is what I don't understand. Any process owns a copy of the shared memory and while copying it doesn't know if another thread will write something to its copy of the shared memory. What if after creating proc1 by calling fork(), another process proc2 is created a little bit later and start writing something into the its copy of the shared memory.
Question: How does proc1 know what to do with the part of the shared memory that are being modified by proc2?
The crucial thing to understand is that there are two different types of memory sharing used.
One is the copy-on-write sharing used by fork() (without exec()), where the child process inherits the parent process's memory and state. In this case when the child or parent modify anything, a new private copy of the modified memory page is allocated. So the child doesn't see changes made by the parent after fork() and the parent doesn't see changes made by the child after fork(). Peer children cannot see each other's changes either. They're all isolated as far as memory is concerned, they just share a common ancestor.
That memory is what's shown in the Program (text), data and stack sections of the diagram.
Because of that isoltion, PostgreSQL also uses POSIX shared memory - or, in older versions, system V shared memory. These are explicitly shared memory segments that are mapped to a range of addresses. Each process sees the same memory, and it is not copy-on-write. It's fully read/write shared.
This is what is shown in the purple "shared memory" section of the diagram.
POSIX shared memory is used for inter-process communication for locking, for shared_buffers, etc etc. Not the memory inherited from fork()ing.
While memory from fork is often shared copy-on-write, that's really an operating system implementation detail. The operating system could choose not to share it at all, and make an immediate copy of the parent's whole address space for the child at fork time. The only way the copy-on-write sharing is really relevant is when looking at top etc.
When PostgreSQL refers to "shared memory" it's always talking about the POSIX or System V shared memory block(s) that are mapped into each process's address space. Not copy-on-write sharing from fork().
I don't know about this special case but generally in linux and most other operating systems in order to speedup creating a new process, when a process asks operating system to create a new process then OS creates the new one with minimum requirements (specifically in DB applications) and share most of parent memory space with child. Now when a child want to modify some part of shared memory, OS uses COW (copy on write) concept and create a new copy of that part of the memory for child process usage. So this part becomes specific for child process and is no longer shared with parent process.
I've learned that a process has the following structure in memory:
(Image from Operating System Concepts, page 82)
However, it is not clear to me what decides that a process looks like this. I guess processes could (and do?) look different if you have a look at non-standard OS / architectures.
Is this structure decided by the OS? By the compiler of the program? By the computer architecture? A combination of those?
Related and possible duplicate: Why do stacks typically grow downwards?.
On some ISAs (like x86), a downward-growing stack is baked in. (e.g. call decrements SP/ESP/RSP before pushing a return address, and exceptions / interrupts push a return context onto the stack so even if you wrote inefficient code that avoided the call instruction, you can't escape hardware usage of at least the kernel stack, although user-space stacks can do whatever you want.)
On others (like MIPS where there's no implicit stack usage), it's a software convention.
The rest of the layout follows from that: you want as much room as possible for downward stack growth and/or upward heap growth before they collide. (Or allowing you to set larger limits on their growth.)
Depending on the OS and executable file format, the linker may get to choose the layout, like whether text is above or below BSS and read-write data. The OS's program loader must respect where the linker asks for sections to be loaded (at least relative to each other, for executables that support ASLR of their static code/data/BSS). Normally such executables use PC-relative addressing to access static data, so ASLRing the text relative to the data or bss would require runtime fixups (and isn't done).
Or position-dependent executables have all their segments loaded at fixed (virtual) addresses, with only the stack address randomized.
The "heap" isn't normally a real thing, especially in systems with virtual memory so each process can have their own private virtual address space. Normally you have some space reserved for the stack, and everything outside that which isn't already mapped is fair game for malloc (actually its underlying mmap(MAP_ANONYMOUS) system calls) to choose when allocating new pages. But yes even modern glibc's malloc on modern Linux does still use brk() to move the "program break" upward for small allocations, increasing the size of "the heap" the way your diagram shows.
I think this is recommended by some committee and then tools like GCC conform to that recommendation. Binary format defines these segments and operating system and its tools facilitate the process of that format to run on system. lets say ELF is recommended by system V and then adopted by unix; and gcc produce the ELF binaries to be run on unix. so i feel story may start from binary format as it decides about memory mappings(code, data/heap/stack). binary format,among other hacks, defines memory mappings to be mapped for loading programs. As for example ELF defines segments (arrange code in text,data,stack to be loaded in memory), GCC generates that segments of ELF binary while loader loads these segments. operating system also has freedom in adjusting the values of these segments like stack size. These are debatable loud thoughts which I try to consolidate.
That figure represents a a specific implementation or an idealized one. A process does not necessarily have that structure. On many systems a process looks only somewhat similar to what is in the diagram.
I am looking forward to understand, what purpose a memory map serves in embedded system.
How does the function stack differs here, from normal unix system.
Any insights that can help me debug few memory related crashes for embedded system will be helpful.
Embedded systems, especially real-time ones, often have a lot of statically-allocated data, and/or data placed at specific locations in memory. The memory map tells you where these things are, which can be helpful when you run into problems and need to examine the state of the system. For example, you might dump all of memory and then analyze it after the fact; in such a case, the memory map will be rather handy for finding the objects you suspect might be related to the problem.
On the code side, your system might log a hardware exception that points to the address of the instruction where the exception was detected. Looking up the memory locations of functions, combined with a disassembly of the function, can help you analyze such problems.
The details really depend on what kind of embedded system you're building. If you provide more details, people may be able to give better responses.
I am not sure that I understand the question. You seem to be suggesting that a "memory map" is something unique to embedded systems or that it is a tangible software component. It is neither; it is merely a description of the layout of an application's memory usage.
All applications will have a memory map regardless of platform, the difference is that typically on an embedded system the application is linked as a single monolithic entity, so that the resultant memory layout refers to the entire system rather than an individual process as it might in an application on a GPOS platform.
It is the linker and the linker script that determines memory mapping, and your linker will be able to output a map report file that describes the layout and allocation applied. This is true of embedded and desktop applications regardless of OS or architecture.
The memory map for a RTOS is not that much different than the memory map for any computer. It defines which hardware resides at which of the processor's addresses. That hardware may be RAM, ROM, Flash, serial ports, parallel ports, timers, interrupt vectors, or any number of other parts addressable by the processor.
The memory map also describes how you intend to budget for limited resources such as RAM, ROM, or Flash in your system design.
For instance, if there's multiple tasks running, RAM might be mapped so that each task has it's own specific area of RAM allocated to it.
In turn, each tasks's part of RAM would be mapped so that there are specific areas for the stack, another for static variables, and perhaps more again for heap(s).
When you have an operating system on the target, it looks after a lot of this dynamically. However, if your application is the only software on the device, you'll have to manage these decisions yourself, usually at compile/link time. Search "link scripts" for further clues,
The Memory map is a layout of memory of system. It is present in both embedded systems and normal applications. Though it is present in normal applications, it's usage is well appreciated in embedded systems due to system constraints.
Memory map is managed by means of linker scripts or linker command files. It maps resources like Flash or Internal RAM(L1P,L1D,L2,L3) or External RAM(DDR) or ROM or peripherals (ports,serial,parallel,USB etc) or specific device registers or I/O ports with appropriate fixed addresses in the memory space of the system.
In case of embedded systems, based on the memory configuration or constraints of board and performance requirements, the segments like text segment or data segment or BSS can also be placed in the appropriate memory of choice.
There are occasions where various versions of development boards will have different configurations of memory and peripherals. In that case, we may need to edit the linker scripts according to memory configuration and peripherals of the board as an essential check-point in board bring-up.
Memory map can help in defining the shared memory too that can play a key role in multi-threaded applications and also for multi-core applications.
Crashes can be debugged by back tracing the address of crash and mapping it to the memory of the system to get an high level idea of the possible library or object causing the problem.