Memory ownership in PKCS #11 C_FindObjects where ulMaxObjectCount != 1 - cryptography

The authors of PKCS #11 v2.40 utilize a common pattern when an API returns a variable length list of items. In APIs such as C_GetSlotList and C_GetMechanismList, the application is expected to call the APIs twice. In the first invocation, a pointer to a CK_ULONG is set to the number of items that will be returned on the next invocation. This allows the application to allocate enough memory and invoke the API again to retrieve the results.
The C_FindObjects call also returns a variable number of items, but it uses a different paradigm. The parameter CK_OBJECT_HANDLE_PTR phObject is set to the head of the result list. The parameter CK_ULONG_PTR pulObjectCount is set to the number of items returned, which is ensured to be less than CK_ULONG ulMaxObjectCount.
The standard does not explicitly say that phObject must be a valid pointer to a block of memory large enough to hold ulMaxObjectCount CK_OBJECT_HANDLEs.
One could interpret the standard as meaning that the application must pessimistically allocate enough memory for ulMaxObjectCount objects. Alternately, one could interpret the standard as meaning that the PKCS #11 implementation will allocate pulObjectCount CK_OBJECT_HANDLEs and it is then the application's responsibility to free that memory. This later interpretation seems suspect however, as no where else in the standard does the implementation of PKCS #11 ever allocate memory.
The passage is:
C_FindObjects continues a search for token and session objects that
match a template, obtaining additional object handles. hSession is
the session’s handle; phObject points to the location that receives
the list (array) of additional object handles; ulMaxObjectCount is
the maximum number of object handles to be returned; pulObjectCount
points to the location that receives the actual number of object
handles returned.
If there are no more objects matching the template, then the location
that pulObjectCount points to receives the value 0.
The search MUST have been initialized with C_FindObjectsInit.
The non-normative example is not very helpful, as it sets ulMaxObjectCount to 1. It does, however, allocate the memory for that one entry. Which seems to indicate that the application must pessimistically pre-allocate the memory.
CK_SESSION_HANDLE hSession;
CK_OBJECT_HANDLE hObject;
CK_ULONG ulObjectCount;
CK_RV rv;
.
.
rv = C_FindObjectsInit(hSession, NULL_PTR, 0);
assert(rv == CKR_OK);
while (1) {
rv = C_FindObjects(hSession, &hObject, 1, &ulObjectCount);
if (rv != CKR_OK || ulObjectCount == 0)
break;
.
.
}
rv = C_FindObjectsFinal(hSession);
assert(rv == CKR_OK);
Specification Link: http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/pkcs11-base-v2.40.pdf

Yes, it would appear that the application is responsible for allocating space for the object handles returned by C_FindObjects(). The example code does this, even though it only requests a single object handle at a time, and so should you.
You could just as well rewrite the example code to request multiple object handles, e.g. like this:
#define MAX_OBJECT_COUNT 100 /* arbitrary value */
K_SESSION_HANDLE hSession;
CK_OBJECT_HANDLE hObjects[MAX_OBJECT_COUNT];
CK_ULONG ulObjectCount, i;
CK_RV rv;
rv = C_FindObjectsInit(hSession, NULL_PTR, 0);
assert(rv == CKR_OK);
while (1) {
rv = C_FindObjects(hSession, hObjects, MAX_OBJECT_COUNT, &ulObjectCount);
if (rv != CKR_OK || ulObjectCount == 0) break;
for (i = 0; i < ulObjectCount; i++) {
/* do something with hObjects[i] here */
}
}
rv = C_FindObjectsFinal(hSession);
assert(rv == CKR_OK);
Presumably, the ability to request multiple object handles in a single C_FindObjects() call is intended as a performance optimization.
FWIW, this is pretty much exactly how many C standard library functions like fread() work as well. It'd be extremely inefficient to read data from a file one byte at a time with fgetc(), so the fread() function lets you allocate an arbitrarily large buffer and read as much data as will fit into it.

Related

Is it safe, to share an array between threads?

Is it safe, to share an array between promises like I did it in the following code?
#!/usr/bin/env perl6
use v6;
sub my_sub ( $string, $len ) {
my ( $s, $l );
if $string.chars > $len {
$s = $string.substr( 0, $len );
$l = $len;
}
else {
$s = $string;
$l = $s.chars;
}
return $s, $l;
}
my #orig = <length substring character subroutine control elements now promise>;
my $len = 7;
my #copy;
my #length;
my $cores = 4;
my $p = #orig.elems div $cores;
my #vb = ( 0..^$cores ).map: { [ $p * $_, $p * ( $_ + 1 ) ] };
#vb[#vb.end][1] = #orig.elems;
my #promise;
for #vb -> $r {
#promise.push: start {
for $r[0]..^$r[1] -> $i {
( #copy[$i], #length[$i] ) = my_sub( #orig[$i], $len );
}
};
}
await #promise;
It depends how you define "array" and "share". So far as array goes, there are two cases that need to be considered separately:
Fixed size arrays (declared my #a[$size]); this includes multi-dimensional arrays with fixed dimensions (such as my #a[$xs, $ys]). These have the interesting property that the memory backing them never has to be resized.
Dynamic arrays (declared my #a), which grow on demand. These are, under the hood, actually using a number of chunks of memory over time as they grow.
So far as sharing goes, there are also three cases:
The case where multiple threads touch the array over its lifetime, but only one can ever be touching it at a time, due to some concurrency control mechanism or the overall program structure. In this case the arrays are never shared in the sense of "concurrent operations using the arrays", so there's no possibility to have a data race.
The read-only, non-lazy case. This is where multiple concurrent operations access a non-lazy array, but only to read it.
The read/write case (including when reads actually cause a write because the array has been assigned something that demands lazy evaluation; note this can never happen for fixed size arrays, as they are never lazy).
Then we can summarize the safety as follows:
| Fixed size | Variable size |
---------------------+----------------+---------------+
Read-only, non-lazy | Safe | Safe |
Read/write or lazy | Safe * | Not safe |
The * indicating the caveat that while it's safe from Perl 6's point of view, you of course have to make sure you're not doing conflicting things with the same indices.
So in summary, fixed size arrays you can safely share and assign to elements of from different threads "no problem" (but beware false sharing, which might make you pay a heavy performance penalty for doing so). For dynamic arrays, it is only safe if they will only be read from during the period they are being shared, and even then if they're not lazy (though given array assignment is mostly eager, you're not likely to hit that situation by accident). Writing, even to different elements, risks data loss, crashes, or other bad behavior due to the growing operation.
So, considering the original example, we see my #copy; and my #length; are dynamic arrays, so we must not write to them in concurrent operations. However, that happens, so the code can be determined not safe.
The other posts already here do a decent job of pointing in better directions, but none nailed the gory details.
Just have the code that is marked with the start statement prefix return the values so that Perl 6 can handle the synchronization for you. Which is the whole point of that feature.
Then you can wait for all of the Promises, and get all of the results using an await statement.
my #promise = do for #vb -> $r {
start
do # to have the 「for」 block return its values
for $r[0]..^$r[1] -> $i {
$i, my_sub( #orig[$i], $len )
}
}
my #results = await #promise;
for #results -> ($i,$copy,$len) {
#copy[$i] = $copy;
#length[$i] = $len;
}
The start statement prefix is only sort-of tangentially related to parallelism.
When you use it you are saying, “I don't need these results right now, but probably will later”.
That is the reason it returns a Promise (asynchrony), and not a Thread (concurrency)
The runtime is allowed to delay actually running that code until you finally ask for the results, and even then it could just do all of them sequentially in the same thread.
If the implementation actually did that, it could result in something like a deadlock if you instead poll the Promise by continually calling it's .status method waiting for it to change from Planned to Kept or Broken, and only then ask for its result.
This is part of the reason the default scheduler will start to work on any Promise codes if it has any spare threads.
I recommend watching jnthn's talk “Parallelism, Concurrency,
and Asynchrony in Perl 6”.
slides
This answer applies to my understanding of the situation on MoarVM, not sure what the state of art is on the JVM backend (or the Javascript backend fwiw).
Reading a scalar from several threads can be done safely.
Modifying a scalar from several threads can be done without having to fear for a segfault, but you may miss updates:
$ perl6 -e 'my $i = 0; await do for ^10 { start { $i++ for ^10000 } }; say $i'
46785
The same applies to more complex data structures like arrays (e.g. missing values being pushed) and hashes (missing keys being added).
So, if you don't mind missing updates, changing shared data structures from several threads should work. If you do mind missing updates, which I think is what you generally want, you should look at setting up your algorithm in a different way, as suggested by #Zoffix Znet and #raiph.
No.
Seriously. Other answers seem to make too many assumptions about the implementation, none of which are tested by the spec.

maxDisplays of CGGetDisplaysWithPoint

Definition:
As defined here, CGGetDisplaysWithPoint takes 4 parameters:
A CGPoint object
An int32 representing the maximum number of displays returned
A mutable array passed by reference, which will be filled with the displayIDs found.
An int32 representing the matching display count
Syntax:
CGError CGGetDisplaysWithPoint(CGPoint point, uint32_t maxDisplays, CGDirectDisplayID *displays, uint32_t *matchingDisplayCount);
This is fine and I can get this function working however I am quite confused as to how I should deal with the maxDisplays parameter?
As I understand it, if I set maxDisplays to 5 then if someone has 6 displays, there is a 1/6 chance that a randomly selected pixel will find no displays?
So do we just set maxDisplays to something unrealistic, like 99, and release the array afterwards? What's the point in this argument?
The point of the argument is to prevent the function from writing past the end of your array. You have to tell it the capacity of the array. Note that the displays parameter is neither a Cocoa nor Core Foundation mutable array object. It's a C-style array. It's "mutable" in the sense that it's not "const", but it's not an object that manages its own storage. You are responsible for managing that storage and must communicate its capacity to any function that is intended to store data in it (or otherwise guarantee that such function won't overrun it).
So, your question should really be how to decide on the capacity of the array. There are two basic approaches:
1) Call the function passing NULL for the displays parameter and any arbitrary value (best to use 0) for maxDisplays. As documented, when displays is NULL, maxDisplays is ignored and the function outputs via matchingDisplayCount the number of displays whose bounds contain the given point. Then, allocate an array with (at least) that many elements to use to receive the display IDs and call the function again, passing that array for displays and its capacity for maxDisplays.
2) Use an array with capacity of 32. It's not explicitly documented but it's implicit in the API that that's the maximum number of supported displays. A display ID can be converted to an OpenGL display mask using CGDisplayIDToOpenGLDisplayMask(). The type CGOpenGLDisplayMask is used to hold OpenGL display masks. It is defined as uint32_t, a 32-bit value. Therefore, there can be at most 32 active displays.
This technique is used in some Apple docs, like here, here, here, and here. That last one even makes a direct connection between the number of bits in CGOpenGLDisplayMask and the maximum number of displays.

Make Realloc behave like Calloc

How can I force Realloc to behave like calloc?
For instance:
I have the following structs:
typedef struct bucket0{
int hashID;
Registry registry;
}Bucket;
typedef struct table0{
int tSize;
int tElements;
Bucket** content;
}Table;
and I have the following code in order to grow the table:
int grow(Table* table){
Bucket** tempPtr;
//grow will add 1 to the number available buckets, and double it.
table->tSize++; //add 1
table->tSize *= 2; //double element
if(!table->content){
//table will be generated for the first time
table->content = (Bucket**)(calloc(sizeof(Bucket*), table->tSize));
} else {
//realloc content
tempPtr = (Bucket**)realloc(table->content, sizeof(Bucket)*table->tSize);
if(tempPtr){
table->content = tempPtr;
return 0;
}else{
return 1000;//table could not grow
}
}
}
When I execute it, the table grows properly, and MOST of the "Buckets" in it are initialized as a NULL ptr. However, not all of them are.
How can I make Realloc behave like calloc? in the sense that when it creates new "buckets" they initialize to NULL
Strictly speaking, you shouldn't be relying on calloc (or memset, for that matter) to set pointers to null. C doesn't guarantee that null pointers are represented by all-zero bytes in memory.
Quoting from the comp.lang.C FAQ question 7.31:
Don't rely on calloc's zero fill too much (see below); usually, it's best to initialize data structures yourself, on a field-by-field basis, especially if there are pointer fields.
calloc's zero fill is all-bits-zero, and is therefore guaranteed to yield the value 0 for all integral types (including '\0' for character types). But it does not guarantee useful null pointer values (see section 5 of this list) or floating-point zero values.
It's safer to initialize the individual structure fields yourself. You can create a static const one as a template, with its content initialized to NULL, and then memcpy it to each element of your dynamically-allocated array.

Variable sized arrays in Objective-C?

Okay, so apparently this works:
void foo(size_t s) {
int myArray[s];
// ... use myArray...
}
Is this really legal? I mean, it must be, because it compiles (where the C compiler would reject it as non-constant). The first part of my question is: how does this work? I assume it's allocating it on the stack? Is this different from using alloca()?
Practically, I found some code that does this:
void bar(size_t chunkSize) {
CFReadStreamRef foo = NULL;
// ...some stuff to init foo...
while (stuffToDo) {
UInt8 buffer[chunkSize];
// ...read some data from stream into buffer
// using CFReadStreamRead()...
}
}
This works. However, when I move the buffer allocation from inside the loop to the first line of the function (directly before foo is declared), the function... stops working. In the debugger it gets to the first access of local variables and then just... exits. I don't see any exceptions being thrown, it doesn't crash, it just program carries on running (in reality the function returns a string and that return value is NULL, which is what the return variable is initialized to). I'm not sure what's going on. The second part of my questions is, in light of the first part, what the heck is going on?
it is legal in C99, although dangerous, and yes -- it is like alloca.
because it's like alloca, you want reasonably sized arrays when allocating on the stack. i am not sure if this is defined if the length is zero, but you could definitely cause a stack overflow if the array is 'large enough' to do so.
as far as what is going on -- pulling it out of the loop should make no difference if the sizes are reasonable. i suspect you are seeing undefined behavior because a parameter value is too large (or perhaps 0) -- you should validate the chunkSize parameter. the assembly will tell you why pulling it out of the loop makes a difference (assuming everything else in the program is well-formed).

passing primitive or struct type as function argument

I'm trying to write some reasonably generic networking code. I have several kinds of packets, each represented by a different struct. The function where all my sending occurs looks like:
- (void)sendUpdatePacket:(MyPacketType)packet{
for(NSNetService *service in _services)
for(NSData *address in [service addresses])
sendto(_socket, &packet, sizeof(packet), 0, [address bytes], [address length]);
}
I would really like to be able to send this function ANY kind of packet, not just MyPacketType packets.
I thought maybe if the function def was:
- (void)sendUpdatePacket:(void*)packetRef
I could pass in anykind of pointer to packet. But, without knowing the type of packet, I can't dereference the pointer.
How do I write a function to accept any kind of primitive/struct as its argument?
What you are trying to achieve is polymorphism, which is an OO concept.
So while this would be quite easy to implement in C++ (or other OO languages), it's a bit more challenging in C.
One way you could get around is it to create a generic "packet" structure such as this:
typedef struct {
void* messageHandler;
int messageLength;
int* messageData;
} packet;
Where the messageHandler member is a function pointer to a callback routine which can process the message type, and the messageLength and messageData members are fairly self-explanatory.
The idea is that the method which you pass the packetStruct to would use the Tell, Don't Ask principle to invoke the specific message handler pointer to by messageHandler, passing in the messageLength and messageData without interpreting it.
The dispatch function (pointed to by messageHandler) would be message-specific and will be able to cast the messageData to the appropriate meaningful type, and then the meaningful fields can be extracted from it and processed, etc.
Of course, this is all much easier and more elegant in C++ with inheritance, virtual methods and the like.
Edit:
In response to the comment:
I'm a little unclear how "able to cast
the messageData to the appropriate
meaningful type, and then the
meaningful fields can be extracted
from it and processed, etc." would be
accomplished.
You would implement a handler for a specific message type, and set the messageHandler member to be a function pointer to this handler. For example:
void messageAlphaHandler(int messageLength, int* messageData)
{
MessageAlpha* myMessage = (MessageAlpha*)messageData;
// Can now use MessageAlpha members...
int messageField = myMessage->field1;
// etc...
}
You would define messageAlphaHandler() in such a way to allow any class to get a function pointer to it easily. You could do this on startup of the application so that the message handlers are registered from the beginning.
Note that for this system to work, all message handlers would need to share the same function signature (i.e. return type and parameters).
Or for that matter, how messageData
would be created in the first place
from my struct.
How are you getting you packet data? Are you creating it manually, reading it off a socket? Either way, you need to encode it somewhere as a string of bytes. The int* member (messageData) is merely a pointer to the start of the encoded data. The messageLength member is the length of this encoded data.
In your message handler callback, you don't want probably don't want to continue to manipulate the data as raw binary/hex data, but instead interpret the information in a meaningful fashion according to the message type.
Casting it to a struct essentially maps the raw binary information on to a meaningful set of attributes matching to the protocol of the message you are processing.
The key is that you must realize that everything in a computer is just an array of bytes (or, words, or double words).
ZEN MASTER MUSTARD is sitting at his desk staring at his monitor staring at a complex pattern of seemingly random characters. A STUDENT approaches.
Student: Master? May I interrupt?
Zen Master Mustard: You have answered your own inquiry, my son.
S: What?
ZMM: By asking your question about interrupting me, you have interrupted me.
S: Oh, sorry. I have a question about moving structures of varying size from place to place.
ZMM: If that it true, then you should consult a master who excels at such things. I suggest, you pay a visit to Master DotPuft, who has great knowledge in moving large metal structures, such as tracking radars, from place to place. Master DotPuft can also cause the slightest elements of a feather-weight strain gage to move with the force of a dove's breath. Turn right, then turn left when you reach the door of the hi-bay. There dwells Master DotPuft.
S: No, I mean moving large structures of varying sizes from place to place in the memory of a computer.
ZMM: I may assist you in that endeavor, if you wish. Describe your problem.
S: Specifically, I have a c function that I want to accept several different types of structs (they will be representing different type of packets). So my struct packets will be passed to my function as void*. But without knowing the type, I can't cast them, or really do much of anything. I know this is a solvable problem, because sento() from socket.h does exactly that:
ssize_t sendto(int socket, const void *message, size_t length, int flags, const struct sockaddr *dest_addr,socklen_t dest_len);
where sendto would be called like:
sendto(socketAddress, &myPacket, sizeof(myPacket), Other args....);
ZMM: Did you describe your problem to Zen Master MANTAR! ?
S: Yeah, he said, "It's just a pointer. Everything in C is a pointer." When I asked him to explain, he said, "Bok, bok, get the hell out of my office."
ZMM: Truly, you have spoken to the master. Did this not help you?
S: Um, er, no. Then I asked Zen Master Max.
ZMM: Wise is he. What was his advice to you useful?
S: No. When I asked him about sendto(), he just swirled his fists in the air. It's just an array of bytes."
ZMM: Indeed, Zen Master Max has tau.
S: Yeah, he has tau, but how do I deal with function arguments of type void*?
ZMM: To learn, you must first unlearn. The key is that you must realize that everything in a computer is just an array of bytes (or, words, or double words). Once you have a pointer to the beginning of a buffer, and the length of the buffer, you can sent it anywhere without a need to know the type of data placed in the buffer.
S: OK.
ZMM: Consider a string of man-readable text. "You plan a tower that will pierce the clouds? Lay first the foundation of humility." It is 82 bytes long. Or, perhaps, 164 if the evil Unicode is used. Guard yourself against the lies of Unicode! I can submit this text to sendto() by providing a pointer to the beginning of the buffer that contains the string, and the length of the buffer, like so:
char characterBuffer[300]; // 300 bytes
strcpy(characterBuffer, "You plan a tower that will pierce the clouds? Lay first the foundation of humility.");
// note that sizeof(characterBuffer) evaluates to 300 bytes.
sendto(socketAddress, &characterBuffer, sizeof(characterBuffer));
ZMM: Note well that the number of bytes of the character buffer is automatically calculated by the compiler. The number of bytes occupied by any variable type is of a type called "size_t". It is likely equivalent to the type "long" or "unsinged int", but it is compiler dependent.
S: Well, what if I want to send a struct?
ZMM: Let us send a struct, then.
struct
{
int integerField; // 4 bytes
char characterField[300]; // 300 bytes
float floatField; // 4 bytes
} myStruct;
myStruct.integerField = 8765309;
strcpy(myStruct.characterField, "Jenny, I got your number.");
myStruct.floatField = 876.5309;
// sizeof(myStruct) evaluates to 4 + 300 + 4 = 308 bytes
sendto(socketAddress, &myStruct, sizeof(myStruct);
S: Yeah, that's great at transmitting things over TCP/IP sockets. But what about the poor receiving function? How can it tell if I am sending a character array or a struct?
ZMM: One way is to enumerate the different types of data that may be sent, and then send the type of data along with the data. Zen Masters refer to this as "metadata", that is to say, "data about the data". Your receiving function must examine the metadata to determine what kind of data (struct, float, character array) is being sent, and then use this information to cast the data back into its original type. First, consider the transmitting function:
enum
{
INTEGER_IN_THE_PACKET =0 ,
STRING_IN_THE_PACKET =1,
STRUCT_IN_THE_PACKET=2
} typeBeingSent;
struct
{
typeBeingSent dataType;
char data[4096];
} Packet_struct;
Packet_struct myPacket;
myPacket.dataType = STRING_IN_THE_PACKET;
strcpy(myPacket.data, "Nothing great is ever achieved without much enduring.");
sendto(socketAddress, myPacket, sizeof(Packet_struct);
myPacket.dataType = STRUCT_IN_THE_PACKET;
memcpy(myPacket.data, (void*)&myStruct, sizeof(myStruct);
sendto(socketAddress, myPacket, sizeof(Packet_struct);
S: All right.
ZMM: Now, just us walk along with the receiving function. It must query the type of the data that was sent and the copy the data into a variable declared of that type. Forgive me, but I forget the exact for of the recvfrom() function.
char[300] receivedString;
struct myStruct receivedStruct;
recvfrom(socketDescriptor, myPacket, sizeof(myPacket);
switch(myPacket.dataType)
{
case STRING_IN_THE_PACKET:
// note the cast of the void* data into type "character pointer"
&receivedString[0] = (char*)&myPacket.data;
printf("The string in the packet was \"%s\".\n", receivedString);
break;
case STRUCT_IN_THE_PACKET:
// note the case of the void* into type "pointer to myStruct"
memcpy(receivedStruct, (struct myStruct *)&myPacket.data, sizeof(receivedStruct));
break;
}
ZMM: Have you achieved enlightenment? First, one asks the compiler for the size of the data (a.k.a. the number of bytes) to be submitted to sendto(). You send the type of the original data is sent along as well. The receiver then queries for the type of the original data, and uses it to call the correct cast from "pointer to void" (a generic pointer), over to the type of the original data (int, char[], a struct, etc.)
S: Well, I'll give it a try.
ZMM: Go in peace.