I'm new to XNU kernel and partially confused.
I see that we a function in file vm_unix.c:
kern_return_t task_for_pid(struct task_for_pid_args *args)
with the comment:
// This should be a BSD system call, not a Mach trap!!!
then 2 declarations in osfmk\mach:
/*
* Obsolete interfaces.
*/
extern kern_return_t task_for_pid(
mach_port_name_t target_tport,
int pid,
mach_port_name_t *t);
extern kern_return_t task_for_pid(
struct task_for_pid_args *args);
and trap decleration in kern
/* 45 */ MACH_TRAP(task_for_pid, 3, 3, munge_www),
So as you can see I found 2 declarations and only 1 implementation.
Where is the implementation of the 3 parameter match trap?
How has the transition from 3 parameters to 1 parameter happened?
MACH_TRAP is a macro which inserts an entry into the mach_trap_table:
#define MACH_TRAP(name, arg_count, u32_arg_words, munge32)
{ (arg_count), (kern_return_t (*)(void *)) (name), (u32_arg_words) }
(see http://newosxbook.com/src.jl?tree=xnu&ver=6153.11.26&file=osfmk/kern/syscall_sw.h)
the arguments are then deserialized in the mach trap invocation, by taking them through the mach_call_munger (http://newosxbook.com/src.jl?tree=xnu&ver=6153.11.26&file=osfmk/i386/bsd_i386.c) which is the wrapper that handles 32/64 bit-ness, and then passes arguments to the actual handler.
Source: "*OS Internals" (http://NewOSXBook.com/) Volume I
Related
2023 update The last person to edit this Q deleted the critically important "LATEST LATEST UPDATE" part that #zentrunix had added near the top. I'm reinstating it.
LATEST LATEST UPDATE
Please see my answer below.
Thanks to everyone who took the time to answer and understand this question.
Original question
Say I have my event-driven TCP communications library in C.
From my Raku application, I can call a function in the C library using NativeCall.
my $server = create-server("127.0.0.1", 4000);
Now, from my callback in C (say onAccept) I want to call out to a Raku function in my application (say on-accept(connection) where connection will be a pointer to a C struct).
So, how can I do that: call my Raku function on-accept from my C function onAccept ?
ps. I tried posting using a simple title "How to call Raku code from C code", but for whatever reason stackoverflow.com wouldn't let me do it. Because of that I concocted this fancy title.
I was creating a 32-bit DLL.
We have to explicitly tell CMake to configure a 64-bit build.
cmake -G "Visual Studio 14 2015 Win64" ..
Anyway, now that the code runs, it's not really what I asked for, because the callback is still in C.
It seems that what I asked for it's not really possible.
I tried to use the approach suggested by Haakon, though I'm afraid I don't understand how it would work.
I'm in Windows, and unfortunately, Raku can't find my dlls, even if I put them in C:\Windows\System32. It finds "msvcrt" (C runtime), but not my dlls.
The dll code (Visual Studio 2015).
#include <stdio.h>
#define EXPORTED __declspec(dllexport)
typedef int (*proto)(const char*);
proto raku_callback;
extern EXPORTED void set_callback(proto);
extern EXPORTED void foo(void);
void set_callback(proto arg)
{
printf("In set_callback()..\n");
raku_callback = arg;
}
void foo(void)
{
printf("In foo()..\n");
int res = raku_callback("hello");
printf("Raku return value: %d\n", res);
}
Cmake code for the
CMAKE_MINIMUM_REQUIRED (VERSION 3.1)
add_library (my_c_dll SHARED my_c_dll.c)
Raku code.
use v6.d;
use NativeCall;
sub set_callback(&callback (Str --> int32))
is native("./my_c_dll"){ * }
sub foo()
is native("./my_c_dll"){ * }
sub callback(Str $str --> Int) {
say "Raku callback.. got string: {$str} from C";
return 32;
}
## sub _getch() returns int32 is native("msvcrt") {*};
## print "-> ";
## say "got ", _getch();
set_callback(&callback);
# foo();
When I run
$ raku test-dll.raku
Cannot locate native library '(null)': error 0xc1
in method setup at D:\tools\raku\share\perl6\core\sources
\947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 298
in block set_callback at D:\tools\raku\share\perl6\core\sources
\947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 594
in block <unit> at test-dll.raku line 21
Raku version.
$ raku -v
This is Rakudo version 2020.05.1 built on MoarVM version 2020.05
implementing Raku 6.d.
Another approach could be to save a callback statically in the C library, for example (libmylib.c):
#include <stdio.h>
static int (*raku_callback)(char *arg);
void set_callback(int (*callback)(char * arg)) {
printf("In set_callback()..\n");
raku_callback = callback;
}
void foo() {
printf("In foo()..\n");
int res = raku_callback("hello");
printf("Raku return value: %d\n", res);
}
Then from Raku:
use v6;
use NativeCall;
sub set_callback(&callback (Str --> int32)) is native('./libmylib.so') { * }
sub foo() is native('./libmylib.so') { * }
sub callback(Str $str --> Int) {
say "Raku callback.. got string: {$str} from C";
return 32;
}
set_callback(&callback);
foo();
Output:
In set_callback()..
In foo()..
Raku callback.. got string: hello from C
Raku return value: 32
Raku is a compiled language; depending on the implementation you've got, it will be compiled to MoarVM, JVM or Javascript. Through compilation, Raku code becomes bytecode in the corresponding virtual machine. So it's never, actually, binary code.
However, Raku code seems to be cleverly organized in a way that an object is actually a pointer to a C endpoint, as proved by Haakon Hagland answer.
WRT to your latest problem, please bear in mind that what you are calling is not a path, but a name that is converted to a navive shared library name and also uses local library path conventions to look for them (it's `PATH' on Windows). So if it's not finding it, add local path to it of simply copy the DLL to one of the searched directories.
First of all, my apologies to #Håkon and #raiph.
Sorry for being so obtuse. :)
Håkon's answer does indeed answer my question, although for whatever reason I have failed to see that until now.
Now the code I played with in order to understand Håkon's solution.
// my_c_dll.c
// be sure to create a 64-bit dll
#include <stdio.h>
#define EXPORTED __declspec(dllexport)
typedef int (*proto)(const char*);
proto raku_function;
extern EXPORTED void install_raku_function(proto);
extern EXPORTED void start_c_processing(void);
void install_raku_function(proto arg)
{
printf("installing raku function\n");
raku_function = arg;
}
void start_c_processing(void)
{
printf("* ----> starting C processing..\n");
for (int i = 0; i < 100; i++)
{
printf("* %d calling raku function\n", i);
int res = raku_function("hello");
printf("* %d raku function returned: %d\n", i, res);
Sleep(1000);
}
}
# test-dll.raku
use v6.d;
use NativeCall;
sub install_raku_function(&raku_function (Str --> int32))
is native("./my_c_dll.dll") { * }
sub start_c_processing()
is native("./my_c_dll.dll") { * }
sub my_raku_function(Str $str --> Int)
{
say "# raku function called from C with parameter [{$str}]";
return 32;
}
install_raku_function &my_raku_function;
start { start_c_processing; }
for ^1000 -> $i
{
say "# $i idling in raku";
sleep 1;
}
$ raku test-dll.raku
installing raku function
# 0 idling in raku
* ----> starting C processing..
* 0 calling raku function
# 0 raku function called from C with parameter [hello]
* 0 raku function returned: 32
# 1 idling in raku
* 1 calling raku function
# 1 raku function called from C with parameter [hello]
* 1 raku function returned: 32
# 2 idling in raku
* 2 calling raku function
# 2 raku function called from C with parameter [hello]
* 2 raku function returned: 32
# 3 idling in raku
* 3 calling raku function
# 3 raku function called from C with parameter [hello]
* 3 raku function returned: 32
# 4 idling in raku
* 4 calling raku function
# 4 raku function called from C with parameter [hello]
* 4 raku function returned: 32
# 5 idling in raku
* 5 calling raku function
# 5 raku function called from C with parameter [hello]
* 5 raku function returned: 32
^CTerminate batch job (Y/N)?
^C
What amazes me is that the Raku signature for my_raku_function maps cleanly to the C signature ... isn't Raku wonderful ? :)
I have been trying to figure this out for a few days now and cannot figure it out. I am using CCS as the IDE and I am working on windows. I am trying to create an RTOS Kernel on a MSP432 and need to use pthreads. I have been able to use pthreads in other examples but I am trying to do my own program and I get this issue when building :
unresolved symbol pthread_create, first referenced in ./armrtk/src/task.obj
I have included the file path into CCS and I cannot use a .cfg file because I am not using XDCTools. I just need help with this and I greatly appreciate it.
I also get a warning:
in pthread_create in TASK.C: #169-D argument of type "void *" is incompatible with parameter of type "void *(*)(void *)"
TASK.H
#ifndef TASK_H
#define TASK_H
#include <pthread.h>
struct task_t {
pthread_t* thread;
int threadCheck;
int state;
};
void *task1(void);
void *task2(void);
struct task_t *create_task(void* functionptr);
void delete_task(void *task);
#endif
TASK.C
#include <task.h>
#include <stdlib.h>
#include <pthread.h>
#define BLOCKED -1
#define READY 0
#define RUNNING 1
int testValue1 = 0;
int testValue2 = 0;
struct task_t *new_task;
pthread_t pntr;
struct task_t *create_task(void* functionptr) {
new_task = malloc(sizeof(struct task_t));
if(!new_task)
return NULL;
//set State of the new thread to ready
new_task->state = 0;
// check to see if pthread is created
**new_task->threadCheck = pthread_create(new_task->thread, NULL, functionptr, NULL);**
if(new_task->threadCheck!= 0){
//thread failed
return NULL;
}
return new_task;
}
void delete_task(void *task) {
if(task != NULL){
free(task);
pthread_exit(NULL);
}
}
The unresolved symbol error is a linker error, not a compiler error. You have failed to link the pthreads library.
With respect to the warning functionptr is a void* where pthread_create() expects a pointer-to-function with signature void fn(void*).
Your task functions have a different signature in any case: void fn(void), so in any event you will need to cast the function pointer in the call to pthread_create() (although you are loosing a useful means of passing information into a task function by omiting the void* argument).
Modify task.h:
typedef void* (*task_t)(void);
struct task_t *create_task( task_t functionptr);
The in task.cpp
new_task->threadCheck = pthread_create( new_task->thread,
NULL,
(void (*)(void *))functionptr,
NULL ) ;
The cast in the pthread_create() call alone would supress the warning, but it bad form to pass a function pointer as a generic void* since it would prevent the compiler warning you if you were to pass anything other then a function pointer of the expected form to to the create_task()`
I have a trivial program:
int main(void)
{
const char sname[]="xxx";
sem_t *pSemaphor;
if ((pSemaphor = sem_open(sname, O_CREAT, 0644, 0)) == SEM_FAILED) {
perror("semaphore initilization");
exit(1);
}
sem_unlink(sname);
sem_close(pSemaphor);
}
When I run it under valgrind, I get the following error:
==12702== Syscall param write(buf) points to uninitialised byte(s)
==12702== at 0x4E457A0: __write_nocancel (syscall-template.S:81)
==12702== by 0x4E446FC: sem_open (sem_open.c:245)
==12702== by 0x4007D0: main (test.cpp:15)
==12702== Address 0xfff00023c is on thread 1's stack
==12702== in frame #1, created by sem_open (sem_open.c:139)
The code was extracted from a bigger project where it ran successfully for years, but now it is causing segmentation fault.
The valgrind error from my example is the same as seen in the bigger project, but there it causes a crash, which my small example doesn't.
I see this with glibc 2.27-5 on Debian. In my case I only open the semaphores right at the start of a long-running program and it seems harmless so far - just annoying.
Looking at the code for sem_open.c which is available at:
https://code.woboq.org/userspace/glibc/nptl/sem_open.c.html
It seems that valgrind is complaining about the line (270 as I look now):
if (TEMP_FAILURE_RETRY (__libc_write (fd, &sem.initsem, sizeof (sem_t)))
== sizeof (sem_t)
However sem.initsem is properly initialised earlier in a fairly baroque manner, firstly by explicitly setting fields in the sem.newsem (part of the union), and then once that is done by a call to memset (L226-228):
/* Initialize the remaining bytes as well. */
memset ((char *) &sem.initsem + sizeof (struct new_sem), '\0',
sizeof (sem_t) - sizeof (struct new_sem));
I think that this particular shenanigans is all quite optimal, but we need to make sure that all of the fields of new_sem have actually been initialised... we find the definition in https://code.woboq.org/userspace/glibc/sysdeps/nptl/internaltypes.h.html and it is this wonderful creation:
struct new_sem
{
#if __HAVE_64B_ATOMICS
/* The data field holds both value (in the least-significant 32 bytes) and
nwaiters. */
# if __BYTE_ORDER == __LITTLE_ENDIAN
# define SEM_VALUE_OFFSET 0
# elif __BYTE_ORDER == __BIG_ENDIAN
# define SEM_VALUE_OFFSET 1
# else
# error Unsupported byte order.
# endif
# define SEM_NWAITERS_SHIFT 32
# define SEM_VALUE_MASK (~(unsigned int)0)
uint64_t data;
int private;
int pad;
#else
# define SEM_VALUE_SHIFT 1
# define SEM_NWAITERS_MASK ((unsigned int)1)
unsigned int value;
int private;
int pad;
unsigned int nwaiters;
#endif
};
So if we __HAVE_64B_ATOMICS then the structure has a data field which contains both the value and the nwaiters, otherwise these are separate fields.
In the initialisation of sem.newsem we can see that these are initialised correctly, as follows:
#if __HAVE_64B_ATOMICS
sem.newsem.data = value;
#else
sem.newsem.value = value << SEM_VALUE_SHIFT;
sem.newsem.nwaiters = 0;
#endif
/* pad is used as a mutex on pre-v9 sparc and ignored otherwise. */
sem.newsem.pad = 0;
/* This always is a shared semaphore. */
sem.newsem.private = FUTEX_SHARED;
I'm doing all of this on a 64-bit system, so I think that valgrind is complaining about the initialisation of the 64-bit sem.newsem.data with a 32-bit value since from:
value = va_arg (ap, unsigned int);
We can see that value is defined simply as an unsigned int which will usually still be 32 bits even on a 64-bit system (see What should be the sizeof(int) on a 64-bit machine?), but that should just be an implicit cast to 64-bits when it is assigned.
So I think this is not a bug - just valgrind getting confused.
I am looking at the pthread_mutex_t structure in the pthreadtypes.h file. What does the "__lock" stand for? Is it like a lock number assigned to the mutex?
typedef union
{
struct __pthread_mutex_s
{
int __lock;
unsigned int __count;
int __owner;
#if __WORDSIZE == 64
unsigned int __nusers;
#endif
/* KIND must stay at this position in the structure to maintain
binary compatibility. */
int __kind;
#if __WORDSIZE == 64
int __spins;
__pthread_list_t __list;
# define __PTHREAD_MUTEX_HAVE_PREV 1
#else
unsigned int __nusers;
__extension__ union
{
int __spins;
__pthread_slist_t __list;
};
#endif
} __data;
char __size[__SIZEOF_PTHREAD_MUTEX_T];
long int __align;
} pthread_mutex_t;
The __lock member of struct __pthread_mutex_s __data is used as a futex object on Linux. Many of the following details may differ depending on the architecture you're looking at:
See the pthread_mutex_lock.c code for the high level locking function for pthread mutexes - __pthread_mutex_lock(), which generally will end up calling LLL_MUTEX_LOCK() and the definitions of LLL_MUTEX_LOCK() and friends, which end up calling lll_lock(), etc., in lowlevellock.h.
The lll_lock() macro in turn calls __lll_lock_wait_private(), which calls lll_futex_wait(), which makes the sys_futex system call.
I have to make tasks as processes in Linux but I don't want the process to execute until all the processes are created. So I thought of moving the processes to wait queue soon after creation and wait until all processes are created.
#include <unistd.h> /* Symbolic Constants */
#include <sys/types.h> /* Primitive System Data Types */
#include <errno.h> /* Errors */
#include <stdio.h> /* Input/Output */
#include <stdlib.h> /* General Utilities */
#include <pthread.h> /* POSIX Threads */
#include <string.h> /* String handling */
#include <sched.h>
#include <linux/kernel.h>
#include <time.h>
#include <sys/resource.h>
#include <stddef.h>
#include <linux/sched.h>
#include <linux/wait.h> /* for wait_event_interruptible() & wake_up_interruptible() */
int done = 0;
static DECLARE_WAIT_QUEUE_HEAD(queue);
int main()
{
int pid1, pid2;
if ((pid1 = fork()) < 0) //create a child process
exit(1);
if ((pid2 = fork()) < 0) //create a child process
exit(1);
if (pid1 == 0) //child process
{
wait_event_interruptible(queue, done == 2);
printf("child 1\n");
}
else //parent process
{
done = done+1;
wake_up_interruptible(&queue);
}
if (pid2 == 0) //child process
{
wait_event_interruptible(queue, done == 2);
printf("child 2\n");
}
else //parent process
{
done = done+1;
wake_up_interruptible(&queue);
}
return 0;
}
But when I tried this sample code it shows these errors.
$ gcc -Wall try.c
try.c:18:8: warning: type defaults to ‘int’ in declaration of ‘DECLARE_WAIT_QUEUE_HEAD’ [-Wimplicit-int]
try.c:18:1: warning: parameter names (without types) in function declaration [enabled by default]
try.c: In function ‘main’:
try.c:33:6: warning: implicit declaration of function ‘wait_event_interruptible’ [-Wimplicit-function-declaration]
try.c:33:31: error: ‘queue’ undeclared (first use in this function)
try.c:33:31: note: each undeclared identifier is reported only once for each function it appears in
try.c:39:2: warning: implicit declaration of function ‘wake_up_interruptible’ [-Wimplicit-function-declaration]
try.c: At top level:
try.c:18:8: warning: ‘DECLARE_WAIT_QUEUE_HEAD’ declared ‘static’ but never defined [-Wunused-function]
When I checked $ man wait_event_interruptible, it says "No manual entry for wait_event_interruptible". So the API is missing in the library. How can I add it to the library? Thanks in advance.
wait_event_interruptible(), wake_up_interruptible() are some of the Kernel's API to create and use wait queues. You cannot use those from the user-land!
If I understand your purpose correctly, what you need to do is to create N processes barrier. If you know the number of the processes (N), you can easily use semaphores: initialize the semaphore with zero, all processes call down() and the last process calls up() N times. You can also use message queues.
You can also use the Linux API for barriers: pthread_barrier_wait and pthread_barrier_init, but I have not used this before.