How does a virtual machine generate native machine code on the fly and execute it?
Assuming you can figure out what are the native machine op-codes you want to emit, how do you go about actually running it?
Is it something as hacky as mapping the mnemonic instructions to binary codes, stuffing it into an char* pointer and casting it as a function and executing?
Or would you generate a temporary shared library (.dll or .so or whatever) and load it into memory using standard functions like LoadLibrary ?
You can just make the program counter point to the code you want to execute. Remember that data can be data or code. On x86 the program counter is the EIP register. The IP part of EIP stands for instruction pointer. The JMP instruction is called to jump to an address. After the jump EIP will contain this address.
Is it something as hacky as mapping the mnemonic instructions to binary codes, stuffing it into an char* pointer and casting it as a function and executing?
Yes. This is one way of doing it. The resulting code would be cast to a pointer to function in C.
Is it something as hacky as mapping the mnemonic instructions to binary codes, stuffing it into an char* pointer and casting it as a function and executing?
Yes, if you were doing it in C or C++ (or something similar), that's exactly what you'd do.
It appears hacky, but that's actually an artifact of the language design. Remember, the actual algorithm you want to use is very simple: determine what instructions you want to use, load them into a buffer in memory, and jump to the beginning of that buffer.
If you really try to do this, though, make sure you get the calling convention right when you return to your C program. I think if I wanted to generate code I'd look for a library to take care of that aspect for me. Nanojit's been in the news recently; you could look at that.
Yup. You just build up a char* and execute it. However, you need to note a couple details. The char* must be in an executable section of memory and must have proper alignment.
In addition to nanojit you can also check out LLVM which is another library that's capable of compiling various program representations down to a function pointer. It's interface is clean and the generated code tends to be efficient.
As far as i know it compiles everything in memory because it has to run some heuristics to to optimize the code (i.e.: inlining over time) but you can have a look at the Shared Source Common Language Infrastructure 2.0 rotor release. The whole codebase is identical to .NET except for the Jitter and the GC.
As well as Rotor 2.0 - you could also take a look at the HotSpot virtual machine in the OpenJDK.
About generating a DLL: the additional required I/O for that, plus linking, plus the complexity of generating the DLL format, would make that much more complicate, and above all they'd kill performance; additionally, in the end you still call a function pointer to the loaded code, so...
Also, JIT compilation can happen one method at a time, and if you want to do that you'd generate lots of small DLLs.
About the "executable section" requirement, calling mprotect() on POSIX systems can fix the permissions (there's a similar API on Win32). You need to do that for a big memory segment instead that once per method since it'd be too slow otherwise.
On plain x86 you wouldn't notice the problem, on x86 with PAE or 64bit AMD64/Intel 64 bit machines you'd get a segfault.
Is it something as hacky as mapping
the mnemonic instructions to binary
codes, stuffing it into an char*
pointer and casting it as a function
and executing?
Yes, that works.
To do this in windows you must set PAGE_EXECUTE_READWRITE to the allocated block:
void (*MyFunc)() = (void (*)()) VirtualAlloc(NULL, sizeofblock, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
//Now fill up the block with executable code and issue-
MyFunc();
Related
Consider a neon register such as:
uint16x8_t foo;
To access an individual lane, one is supposed to use vgetq_lane_u16(foo, 3). However, one might be tempted to write foo[3] instead given the intuition that foo is an array of shorts. When doing so, the gcc (10) compiles without warnings, but it is not clear that it does what it was intended to do.
The gcc documentation does not specifically mention the indexed access, but it says that operations behave like C++ valarrays. Those do support indexed access in the intuitive way.
Regardless of what foo[3] evaluates to, doing so seems to be faster than vgetq_lane_u16(foo, 3), so probably they're different or we wouldn't need both.
So what exactly does foo[3] mean? Is its behavior defined at all? If not, why does gcc happily compile it?
The foo[3] form is the GCC Vector extension form, as you have found and linked documentation to; it behaves as so:
Vectors can be subscripted as if the vector were an array with the same number of elements and base type. Out of bound accesses invoke undefined behavior at run time. Warnings for out of bound accesses for vector subscription can be enabled with -Warray-bounds.
This can have surprising results when used on big-endian systems, so Arm’s Arm C Language Extensions recommend to use vget_lane if you are using other Neon intrinsics like vld1 in the same code path.
Inside a KEXT, I need to do some processing where I would get either a proc_t or just a pid.
If I go the pid route, I would do a sysctl() of sorts.
Unfortunately, I can't do either. proc_t is undefined and sysctl() isn't either. sysctlbyname() can be called but kinfo_proc isn't defined. If I try to use proc_t, the compiler complains about forward definition of [struct proc]
I'm assuming that sysctl() is there to be used in user mode but is there any way I can use proc_t?
I tried to use the XNU/osfmk/bsd include dir but then it won't compile because of redefinitions and other errors.
It's a little disconcerting and I'm still trying to wrap my head around what I can and cannot do.
Surely this can be done but I just don't quite know how.
OK, I'm going to try and take a stab at the question I think you're asking.
As you've discovered, a proc_t is a pointer to an opaque struct proc. Don't write it off though, as there are various functions that operate on such pointers, so you don't need to gain direct access to the struct (which helps maintain binary compatibility). Most of these are declared in sys/proc.h in the Kernel.framework - i.e. /System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/proc.h. You mention PID and parent PID, for which there are the following:
/* returns the pid of the given process */
extern int proc_pid(proc_t);
/* returns the pid of the parent of a given process */
extern int proc_ppid(proc_t);
There are also functions for going the other way - getting the proc_t for a PID etc.
Note that these functions are part of the BSD portion of the kernel, so your kext needs to declare a dependency on the BSD KPI bundle in its info.plist. (look up the kextlibs tool if you haven't come across this yet)
Coming from Windows, you'll probably have to get used to reading header files and source codes instead of documentation. Much of the OSX kernel API is undocumented.
In my VB6 application I make several calls to a COM server my team created from a Ada project (using GNATCOM). There are basically 2 methods available on the COM server. Their prototypes in VB are:
Sub PutParam(Param As Parameter_Type, Value)
Function GetParam(Param As Parameter_Type)
where Parameter_Type is an enumerated type which distinguishes the many parameters I can put to/get from the COM server and 'Value' is a Variant type variable. PutParam() receives a variant and GetParam() returns a variant. (I don't really know why in the VB6 Object Browser there's no reference to the Variant type on the COM server interface...).
The product of this project has been used continuously this way for years without any problems in this interface on computers with Windows XP with SP2. On computers with WinXP SP3 we get the error 0x800706F7 "The stub received bad data" when trying to put parameters with the 'Long' type.
Does anybody have any clue on what could be causing this? The COM server is still being built in a system with SP2. Should make any difference building it on a system with SP3? (like when we build for X64 in X64 systems).
One of the calls that are causing the problem is the following (changed some var names):
Dim StructData As StructData_Type
StructData.FirstLong = 1234567
StructData.SecondLong = 8901234
StructData.Status = True
ComServer.PutParam(StructDataParamType, StructData)
Where the definition of StructData_Type is:
Type StructData_Type
FirstLong As Long
SecondLong As Long
Status As Boolean
End Type
(the following has been added after the question was first posted)
The definition of the primitive calls on the interface of the COM server in IDL are presented below:
// Service to receive data
HRESULT PutParam([in] Parameter_Type Param, [in] VARIANT *Value);
//Service to send requested data
HRESULT GetParam([in] Parameter_Type Param, [out, retval] VARIANT *Value);
The definition of the structure I'm trying to pass is:
struct StructData_Type
{
int FirstLong;
int SecondLong;
VARIANT_BOOL Status;
} StructData_Type;
I found it strange that this definition here is using 'int' as the type of FirstLong and SeconLong and when I check the VB6 object explorer they are typed 'Long'. Btw, when I do extract the IDL from the COM server (using a specific utility) those parameters are defined as Long.
Update:
I have tested the same code with a version of my COM server compiled for Windows 7 (different version of GNAT, same GNATCOM version) and it works! I don't really know what happened here. I'll keep trying to identify the problem on WinXP SP3 but It is good to know that it works on Win7. If you have a similar problem it may be good to try to migrate to Win7.
I'll focus on explaining what the error means, there are too few hints in the question to provide a simple answer.
A "stub" is used in COM when you make calls across an execution boundary. It wasn't stated explicitly in the question but your Ada program is probably an EXE and implements an out-of-process COM server. Crossing the boundary between processes in Windows is difficult due to their strong isolation. This is done in Windows by RPC, Remote Procedure Call, a protocol for making calls across such boundaries, a network being the typical case.
To make an RPC call, the arguments of a function must be serialized into a network packet. COM doesn't know how to do this because it doesn't know enough about the actual arguments to a function, it needs the help of a proxy. A piece of code that does know what the argument types are. On the receiving end is a very similar piece of code that does the exact opposite of what the proxy does. It deserializes the arguments and makes the internal call. This is the stub.
One way this can fail is when the stub receives a network packet and it contains more or less data than required for the function argument values. Clearly it won't know what to do with that packet, there is no sensible way to turn that into a StructData_Type value, and it will fail with "The stub received bad data" error.
So the very first explanation for this error to consider is a DLL Hell problem. A mismatch between the proxy and the stub. If this app has been stable for a long time then this is not a happy explanation.
There's another aspect about your code snippet that is likely to induce this problem. Structures are very troublesome beasts in software, their members are aligned to their natural storage boundary and the alignment rules are subject to interpretation by the respective compilers. This can certainly be the case for the structure you quoted. It needs 10 bytes to store the fields, 4 + 4 + 2 and they align naturally. But the structure is actually 12 bytes long. Two bytes are padded at the end to ensure that the ints still align when the structure is stored in an array. It also makes COM's job very difficult, since COM hides implementation detail and structure alignment is a massive detail. It needs help to copy a structure, the job of the IRecordInfo interface. The stub will also fail when it cannot find an implementation of that interface.
I'll talk a bit about the proxy, stub and IRecordInfo. There are two basic ways a proxy/stub pair are generated. One way is by describing the interfaces in a language called IDL, Interface Description Language, and compile that with MIDL. That compiler is capable of auto-generating the proxy/stub code, since it knows the function argument types. You'll get a DLL that needs to be registered on both the client and the server. Your server might be using that, I don't know.
The second way is what VB6 uses, it takes advantage of a universal proxy that's built into Windows. Called FactoryBuffer, its CLSID is {00000320-0000-0000-C000-000000000046}. It works by using a type library. A type library is a machine readable description of the functions in a COM server, good enough for FactoryBuffer to figure out how to serialize the function arguments. This type library is also the one that provides the info that IRecordInfo needs to figure out how the members of a structure are aligned. I don't know how it is done on the server side, never heard of GNATCOM before.
So a strong explanation for this problem is that you are having a problem with the type library. Especially tricky in VB6 because you cannot directly control the guids that it uses. It likes to generate new ones when you make trivial changes, the only way to avoid it is by selecting the binary compatibility option. Which uses an old copy of the type library and tries to keep the new one as compatible as possible. If you don't have that option turned on then do expect trouble, especially for the guid of the structure. Kaboom if it changed and the other end is still using the old guid.
Just some hints on where to start looking. Do not assume it is a problem caused by SP3, this COM infrastructure hasn't changed for a very long time. But certainly expect this kind of problem due to a new operating system version being installed and having to re-register everything. SysInternals' ProcMon is a good utility to see the programs use the registry to find the proxy, stub and type library. And you'd certainly get help from a COM Spy kind of utility, albeit that they are very hard to find these days.
If it suddenly stopped working happily on XP, the first culprit I'd look for is type mismatches. It is possible that "long" on such systems is now 64-bits, while your Ada COM code (and/or perhaps your C ints) are exepecting 32-bits. With a traditionally-compiled system this would have been checked for you by your compiler, but the extra indirection you have with COM makes that difficult.
The bit you wrote in there about "when we compile for 64-bit systems" makes me particularly leery. 64-bit compiles may change the size of many C types, you know.
This Related Post suggests you need padding in your struct, as marshalling code may expect more data than you actually send (which is a bug, of course). Your struct contains 9 bytes (assuming 4 bytes for each of the ints/longs and one for the boolean). Try to add padding so that your struct contains a multiple of 4 bytes (or, failing that, multiple of 8, as the post isn't clear on the expected size)
I am also suggesting that the problem is due to a padding issue in your structure. I don't know whether you can control this using a #pragma, but it might be worth looking at your documentation.
I think it would be a good idea to try and patch your struct so that the resulting type library struct is a multiple of four (or eight). Your Status member takes up 2 bytes, so maybe you should insert a dummy value of the same type either before or after Status - which should bring it up to 12 bytes (if packing to eight bytes, this would have to be three dummy variables).
I could not find any good document on internet about STM32 programming. STM's own documents do not explain anything more than register functions. I will greatly appreciate if anyone can explain my following questions?
I noticed that in all example programs that STM provides, local variables for main() are always defined outside of the main() function (with occasional use of static keyword). Is there any reason for that? Should I follow a similar practice? Should I avoid using local variables inside the main?
I have a gloabal variable which is updated within the clock interrupt handle. I am using the same variable inside another function as a loop condition. Don't I need to access this variable using some form of atomic read operation? How can I know that a clock interrupt does not change its value in the middle of the function execution? Should I need to cancel clock interrupt everytime I need to use this variable inside a function? (However, this seems extremely ineffective to me as I use it as loop condition. I believe there should be better ways of doing it).
Keil automatically inserts a startup code which is written in assembly (i.e. startup_stm32f4xx.s). This startup code has the following import statements:
IMPORT SystemInit
IMPORT __main
.In "C", it makes sense. However, in C++ both main and system_init have different names (e.g. _int_main__void). How can this startup code can still work in C++ even without using "extern "C" " (I tried and it worked). How can the c++ linker (armcc --cpp) can associate these statements with the correct functions?
you can use local or global variables, using local in embedded systems has a risk of your stack colliding with your data. with globals you dont have that problem. but this is true no matter where you are, embedded microcontroller, desktop, etc.
I would make a copy of the global in the foreground task that uses it.
unsigned int myglobal;
void fun ( void )
{
unsigned int myg;
myg=myglobal;
and then only use myg for the rest of the function. Basically you are taking a snapshot and using the snapshot. You would want to do the same thing if you are reading a register, if you want to do multiple things based on a sample of something take one sample of it and make decisions on that one sample, otherwise the item can change between samples. If you are using one global to communicate back and forth to the interrupt handler, well I would use two variables one foreground to interrupt, the other interrupt to foreground. yes, there are times where you need to carefully manage a shared resource like that, normally it has to do with times where you need to do more than one thing, for example if you had several items that all need to change as a group before the handler can see them change then you need to disable the interrupt handler until all the items have changed. here again there is nothing special about embedded microcontrollers this is all basic stuff you would see on a desktop system with a full blown operating system.
Keil knows what they are doing if they support C++ then from a system level they have this worked out. I dont use Keil I use gcc and llvm for microcontrollers like this one.
Edit:
Here is an example of what I am talking about
https://github.com/dwelch67/stm32vld/tree/master/stm32f4d/blinker05
stm32 using timer based interrupts, the interrupt handler modifies a variable shared with the foreground task. The foreground task takes a single snapshot of the shared variable (per loop) and if need be uses the snapshot more than once in the loop rather than the shared variable which can change. This is C not C++ I understand that, and I am using gcc and llvm not Keil. (note llvm has known problems optimizing tight while loops, very old bug, dont know why they have no interest in fixing it, llvm works for this example).
Question 1: Local variables
The sample code provided by ST is not particularly efficient or elegant. It gets the job done, but sometimes there are no good reasons for the things they do.
In general, you use always want your variables to have the smallest scope possible. If you only use a variable in one function, define it inside that function. Add the "static" keyword to local variables if and only if you need them to retain their value after the function is done.
In some embedded environments, like the PIC18 architecture with the C18 compiler, local variables are much more expensive (more program space, slower execution time) than global. On the Cortex M3, that is not true, so you should feel free to use local variables. Check the assembly listing and see for yourself.
Question 2: Sharing variables between interrupts and the main loop
People have written entire chapters explaining the answers to this group of questions. Whenever you share a variable between the main loop and an interrupt, you should definitely use the volatile keywords on it. Variables of 32 or fewer bits can be accessed atomically (unless they are misaligned).
If you need to access a larger variable, or two variables at the same time from the main loop, then you will have to disable the clock interrupt while you are accessing the variables. If your interrupt does not require precise timing, this will not be a problem. When you re-enable the interrupt, it will automatically fire if it needs to.
Question 3: main function in C++
I'm not sure. You can use arm-none-eabi-nm (or whatever nm is called in your toolchain) on your object file to see what symbol name the C++ compiler assigns to main(). I would bet that C++ compilers refrain from mangling the main function for this exact reason, but I'm not sure.
STM's sample code is not an exemplar of good coding practice, it is merely intended to exemplify use of their standard peripheral library (assuming those are the examples you are talking about). In some cases it may be that variables are declared external to main() because they are accessed from an interrupt context (shared memory). There is also perhaps a possibility that it was done that way merely to allow the variables to be watched in the debugger from any context; but that is not a reason to copy the technique. My opinion of STM's example code is that it is generally pretty poor even as example code, let alone from a software engineering point of view.
In this case your clock interrupt variable is atomic so long as it is 32bit or less so long as you are not using read-modify-write semantics with multiple writers. You can safely have one writer, and multiple readers regardless. This is true for this particular platform, but not necessarily universally; the answer may be different for 8 or 16 bit systems, or for multi-core systems for example. The variable should be declared volatile in any case.
I am using C++ on STM32 with Keil, and there is no problem. I am not sure why you think that the C++ entry points are different, they are not here (Keil ARM-MDK v4.22a). The start-up code calls SystemInit() which initialises the PLL and memory timing for example, then calls __main() which performs global static initialisation then calls C++ constructors for global static objects before calling main(). If in doubt, step through the code in the debugger. It is important to note that __main() is not the main() function you write for your application, it is a wrapper with different behaviour for C and C++, but which ultimately calls your main() function.
Is it valid to develop a DLL in C++ that returns boost shared pointers and uses them as parameters?
So, is it ok to export functions like this?
1.) boost::shared_ptr<Connection> startConnection();
2.) void sendToConnection(boost::shared_ptr<Connection> conn, byte* data, int len);
In special: Does the reference count work across DLL boundaries or would the requirement be that exe and dll use the same runtime?
The intention is to overcome the problems with object ownership. So the object gets deleted when both dll and exe don't reference it any more.
According to Scott Meyers in Effective C++ (3rd Edition), shared_ptrs are safe across dll boundaries. The shared_ptr object keeps a pointer to the destructor from the dll that created it.
In his book in Item 18 he states, "An especially nice feature of
tr1::shared_ptr is that it automatically uses its per-pointer deleter
to eliminate another potential client error, the "cross-DLL problem."
This problem crops up when an object is created using new in one
dynamically linked library (DLL) but is deleted in a different DLL. On
many platforms, such cross-DLL new/delete pairs lead to runtime
errors. tr1::shared_ptr avoid the problem, because its default deleter
uses delete from the same DLL where the tr1::shared_ptr is created."
Tim Lesher has an interesting gotcha to watch for, though, that he mentions here. You need to make sure that the DLL that created the shared_ptr isn't unloaded before the shared_ptr finally goes out of scope. I would say that in most cases this isn't something you have to watch for, but if you're creating dlls that will be loosely coupled then I would recommend against using a shared_ptr.
Another potential downside is making sure both sides are created with compatible versions of the boost library. Boost's shared_ptr has been stable for a long while. At least since 1.34 it's been tr1 compatible.
In my opinion, if it's not in the standard and it's not an object/mechanism provided by your library, then it shouldn't be part of the interface to the library. You can create your own object to do the reference counting, and perhaps use boost underneath, but it shouldn't be explicitly exposed in the interface.
DLLs do not normally own resources - the resources are owned by the processes that use the DLL. You are probably better off returning a plain pointer, which you then store in a shared pointer on the calling side. But without more info it's hard to be 100% certain about this.
Something to lookout for if you expose raw pointers from a dll interface. It forces you to use the shared dll CRT, memory allocated in one CRT cannot be deallocated in a different CRT. If you use the shared dll CRT in all your modules ( dll's & exe's ) then you are fine, they all share the same heap, if you dont you will be crossing CRT's and the world will meltdown.
Aside from that issue, I agree with the accepted answer. The creation factory probably shouldn't define ownership & lifecycle management for the client code.
No it is not.
The layout of boost::shared_ptr<T> might not be the same on both sides of the DLL boundary. (Layout is influenced by compiler version, packing pragmas, and other compiler options, as well as the actual version of the Boost source code.)
Only "standard layout" (a new concept in C++11, related to the old "POD = plain old data" concept) types can safely be passed between separately-built modules.