I need to watch a folder for files- files will drop into folders, and it may take several seconds or a few minutes for the file copy to be complete. I've read multiple topics on SO (Checking File sizes for changes, Detect file in use by other process). Neither of these give a great answer.
Polling is "bad", but how can I know if a file stops increasing in size? Specifically, is there a notification for "file size is constant" or "file is complete"? Can the OS notify of non-activity (IOW, how do you prove a negative?). It would seem to me that logically, one MUST poll a file to see if it's not changing. I've also checked SCEvents and UKKQueue, but again both only notify of a change. UKKQueue has a "file size increased" method, but no "file size has not increased method".
Is there really any way to detect file copy completion without polling or using sleep()?
This is the code I used to monitor file locally. I am not sure if this would work for you.
int fileHander = open("/location/file", O_RDONLY);
dispatch_queue_t queue = dispatch_get_global_queue(0, 0);
unsigned long mask = DISPATCH_VNODE_DELETE | DISPATCH_VNODE_WRITE | DISPATCH_VNODE_EXTEND | DISPATCH_VNODE_ATTRIB | DISPATCH_VNODE_LINK | DISPATCH_VNODE_RENAME | DISPATCH_VNODE_REVOKE;
__block dispatch_source_t source;
void (^changeHandler)(void) = ^{
unsigned long l = dispatch_source_get_data(source);
if (l & DISPATCH_VNODE_DELETE) {
printf("file deleted");
dispatch_source_cancel(source);
}
else {
printf("file data changed");
}
};
void (^cancelHandler)(void) = ^{
int fileHander = dispatch_source_get_handle(source);
close(fileHander);
};
source = dispatch_source_create(DISPATCH_SOURCE_TYPE_VNODE,fileHander, mask, queue);
dispatch_source_set_event_handler(source, changeHandler);
dispatch_source_set_cancel_handler(source, cancelHandler);
dispatch_resume(source);
If you have control over the files being delivered, copy them into a temporary directory on the same volume, and when the copy is done, then move the file. The move simply relinks the file in the file system. Then, kqueue can notify you when the file is present. It's presence means the whole file is there.
BTW, when you use the "atomically" version of the cocoa file manager API, this is pretty much what it does behind the scenes.
If you don't have control, and you just want to monitor files, then use kqueue to notify you when a file shows up. As the file grows, queue will notify you that it has been extended. However, it does not know if some other app is done extending the file or not, so you still have to have some sort of timer to check for change.
I would kick off an interval timer at the same time I register for kqueue NOTE_EXTEND events. I would keep track of the last time I saw a NOTE_EXTEND event, and if I had not seen one since the last timer fired, I would assume the file has stopped being extended.
Now, you have to determine what timer value to use... and if you want to back off and keep looking for a "while" but unless the application copying the file does so via a "move" or unless it sends a notification that the file has been fully copied, you are going to have to do some type of timer, with some arbitrary value... at which time you assume it's done.
Obviously, if you can fit into the first option, things are much better as you have a deterministic way of knowing that the file has stopped growing.
Related
I have seen that there are quite a few questions about jumping from an app to the ST system bootloader, for example this one. These use the method of setting the MSP and PC then doing the jump with a function pointer.
This seems to cause an issue with the system bootloader dual-bank management whereby the first jump fails and a second jump needs to be done.
My question is - would it be possible/better to use the user option bytes to jump to the bootloader instead?
Since the OB register is read during boot in the OBL phase, if we set both the "nBOOT1 bit" and "nBOOT_SEL bit" and clear the "nBOOT0 bit" then do a soft reset would this avoid the empty check weirdness and let us jump to the bootloader in one go?
(Just for context - this would be the first step of doing updates via CAN as the MCU in question has a CAN bootloader built in)
Thanks in advance!
After some time tinkering with a dev board and with some help from Tilen Majerle I found that this is indeed possible and does work well.
I added the following in my main() while(1) loop so that when the blue button is pressed, the user option bits are modified and a reset is performed.
I found that we don't have to do the soft reset ourselves as the HAL_FLASH_OB_Launch() function triggers the reset for us, after which we should boot into system memory according to the reference manual page 67.
Also I found that the flash and option bytes must be unlocked before setting the option bytes, but not locked afterwards or the reset won't occur.
Here is the code to do it:
if(HAL_GPIO_ReadPin(BUTTON_GPIO_Port, BUTTON_Pin) == GPIO_PIN_RESET)
{
// Basic de-bounce for testing
HAL_Delay(100);
while(HAL_GPIO_ReadPin(BUTTON_GPIO_Port, BUTTON_Pin) == GPIO_PIN_RESET)
{
__NOP();
}
// Read, modify & write user option bits
// nBOOT1 = 1, nBOOT_SEL = 1, nBOOT0 = 0; will select system memory as boot area
uint32_t optBits = FLASH->OPTR;
optBits = (optBits | FLASH_OPTR_nBOOT1 | FLASH_OPTR_nBOOT_SEL);
optBits &= ~(FLASH_OPTR_nBOOT0);
// Unlock flash
HAL_FLASH_Unlock();
// Clear OPTLOCK
HAL_FLASH_OB_Unlock();
// Set up struct with desired bits
FLASH_OBProgramInitTypeDef optionBytesSetting = {0};
optionBytesSetting.OptionType = OPTIONBYTE_USER;
optionBytesSetting.USERConfig = optBits;
optionBytesSetting.USERType = OB_USER_nBOOT0;
// Write Option Bytes
HAL_FLASHEx_OBProgram(&optionBytesSetting);
HAL_Delay(10);
// Soft reset
HAL_FLASH_OB_Launch();
NVIC_SystemReset(); // is not reached
}
I verified that the flash OPTR register is modified correctly (it goes from 0xFFFFFEAA to 0xFBFFFEAA, essentially just the nBOOT0 bit is cleared as the other two bits were already set). The MCU does reset at HAL_FLASH_OB_Launch() as expected and pausing the program reveals that after reset it is running the system bootloader based on the PC address.
I also verified it using STM32CubeProgrammer which allows me to view the PC and option bytes, plus lets me set nBOOT0 back to 1 and boot the board to my app.
As for reverting the OB settings programmatically, you could either use the Write Memory command before jumping to the app, or you could use the Go command to jump to the app then modify the option bytes first thing in your app.
Older, now deprecated, macOS file system APIs provided flags to read a file unbuffered.
I seek a modern way to accomplish the same, so that I can read a file's data into memory without it being cached needlessly somewhere else in memory (such as the volume cache).
Reading with fread and first calling setvbuf (fp, NULL, _IONBF, 0) is not having the desired effect in my tests, for example. I am seeking other low-level functions that let me read into a prepared memory buffer and that let me avoid buffering of the whole data.
Background
I am writing a file search program. It reads large amounts of file content (many GBs) that isn't and won't be used by the user otherwise. It would be a waste to have all this data cached in the volume cache as it'll soon get purged by further reads again, anyway. It'll also likely lead to purging file data that's actually in use by the user or system, causing more cache misses.
Therefore, I should be able to tell the system that I do not need the file data cached. The little caching needed for cluster boundaries is not an issue. It's the many large chunks that I read briefly into memory to search it that is not needed to be cached.
Two suggestions:
Use the read() system call instead of stdio.
Disable data caching with the F_NOCACHE option for fcntl().
In Swift that would be something like (error checking omitted for brevity):
import Foundation
let path = "/path/to/file"
let fd = open(path, O_RDONLY)
fcntl(fd, F_NOCACHE, 1)
var buffer = Data(count: 1024 * 1024)
buffer.withUnsafeMutableBytes { ptr in
let amount = read(fd, ptr.baseAddress, ptr.count)
}
close(fd)
I'm setting up a market data back-testing using Chronicle Queue (CQ), reading data from a binary file then writing into a single CQ and simultaneously reading the data from that CQ and dumping the statistics. I am doing a POC to replace our existing real-time market data feed handler worker queue.
While doing basic read/writes testing on Linux/SSD setup, I see reads are lagging behind writes - in fact latency is accumulating. Both Appender and Tailer are running as separate processes on same host.
Would like to know, if there is any issue in the code I am using?
Below is the code snippet -
Writer -
In constructor -
myQueue = SingleChronicleQueueBuilder.binary(queueName).build();
myAppender = myQueue.acquireAppender();
In data callback -
myAppender.writeDocument(myDataPacket);
myQueue.close();
where myDataPacket is Java object wrapping the byte[] and other fields.
Tailer -
In Constructor -
myQueue = SingleChronicleQueueBuilder.binary(queueName).build();
myTailer = myQueue.createTailer();
In Read method -
while (notLastRecord)
{
if(myTailer.readDocument(myDataPacket))
{
notLastRecord = ;
//do stuff
}
}
myQueue.close();
Any help is highly appreciated.
Thanks,
Pavan
First of all I assume by "reads are lagging behind writes - in fact latency is accumulating" you mean that for every every subsequent message, the time the message is read from the queue is further from the time the event was written to the queue.
If you see latency accumulating like that, most likely the data is produced much quicker then you can consume it which from the use case you described is very much possible - if all you need at the write side is parsing simple text line and dump it into a queue file, it's quick, but if you do some processing when you read the entry from the queue - it might be slower.
From the code it's not clear what/how much work your code is doing, and the code looks OK to me, except you probably shouldn't call queue.close() after each appender.writeDocument() call but most likely you are not doing this otherwise it would blow up.
Without seeing actual code or test case it's impossible to say more.
I have an array of objects that each needs to load itself from binary file data. I create an array of these objects and then call an AsyncAction for each of them that starts it reading in its data. Trouble is, they are not loading entirely - they tend to get only part of the data from the files. How can I make sure that the whole thing is read? Here is an outline of the code: first I enumerate the folder contents to get a StorageFile for each file it contains. Then, in a for loop, each receiving object is created and passed the next StorageFile, and it creates its own Buffer and DataReader to handle the read. m_file_bytes is a std::vector.
m_buffer = co_await FileIO::ReadBufferAsync(nextFile);
m_data_reader = winrt::Windows::Storage::Streams::DataReader::FromBuffer(m_buffer);
m_file_bytes.resize(m_buffer.Length());
m_data_reader.ReadBytes(m_file_bytes);
My thought was that since the buffer and reader are class members of the object they would not go out of scope and could finish their work uninterrupted as the next objects were asked to load themselves in separate AsyncActions. But the DataReader only gets maybe half of the file data or less. What can be done to make sure it completes? Thanks for any insights.
[Update] Perhaps what is going is that the file system can handle only one read task at a time, and by starting all these async reads each is interrupting the previous one -? But there must be a way to progressively read a folder full of files.
[Update] I think I have it working, by adopting the principle of concentric loops - the idea is not to proceed to the next load until the first one has completed. I think - someone can correct me if I'm wrong, that the file system cannot do simultaneous reads. If there is an accepted and secure example of how to do this I would still love to hear about it, so I'm not answering my own question.
#include <wrl.h>
#include <robuffer.h>
uint8_t* GetBufferData(winrt::Windows::Storage::Streams::IBuffer& buffer)
{
::IUnknown* unknown = winrt::get_unknown(buffer);
::Microsoft::WRL::ComPtr<::Windows::Storage::Streams::IBufferByteAccess> bufferByteAccess;
HRESULT hr = unknown->QueryInterface(_uuidof(::Windows::Storage::Streams::IBufferByteAccess), &bufferByteAccess);
if (FAILED(hr))
return nullptr;
byte* bytes = nullptr;
bufferByteAccess->Buffer(&bytes);
return bytes;
}
https://learn.microsoft.com/en-us/cpp/cppcx/obtaining-pointers-to-data-buffers-c-cx?view=vs-2017
https://learn.microsoft.com/en-us/windows/uwp/xbox-live/storage-platform/connected-storage/connected-storage-using-buffers
I'm trying to use the "top" command in MacOS X to determine which app is using the resources.
When I do:
top -stats "pid,command"
the command column is truncated, if the process name is too long.
if you look at the activity monitor, the process name is shown properly (with full name) + icon. My questions are:
how to get the full process name?
sometimes the app icon show next to the process name, is there anyway to do the similar thing using objective-c? should I simply navigate to the app contents folder and grab the icns image?
First, if you're trying to get the data programmatically, driving top is almost definitely not what you want to do.
But, to answer your direct questions:
how to get the full process name?
There is no way to control the truncation of commands. You can use the -ncols parameter to set the width of the output for non-interactive output, but that doesn't stop top from truncating if it wants to.
sometimes the app icon show next to the process name, is there anyway to do the similar thing using objective-c? should I simply navigate to the app contents folder and grab the icns image?
No. How would you deal with apps that have multiple .icns files, e.g., for document icons? (Try it with iTunes, for example. If you pick the first .icns, you get the AIFF document icon; if you pick the last, you get the internal-use recent TV shows icon.)
The right way to do it is to get the NSBundle for the application, then do something like this:
NSString *iconFile = [bundle objectForInfoDictionaryKey:#"CFBundleIconFile"];
if (iconFile) {
NSString *iconPath = [bundle pathForResource:iconFile ofType:#"icns"];
// load and display the icon
}
So, how do you actually want to do this, if not by driving top?
Well, what you're asking for is actually not a well-defined thing. OS X has four different notions of task/process/program/application that don't correspond 1-to-1, and that makes life difficult if you want to write a mashup of two programs that use different notions—e.g., top deals in BSD processes, while Activity Monitor deals in OS X applications.
If what you actually want is the same list top uses, it's open source, so you can read it and do the same thing it does.
But the simplest way to get the list of BSD processes is probably the interfaces in libproc.h, in particular proc_listallpids and proc_pidinfo. For example:
int dump_proc_names() {
int buf[16384];
int count = proc_listallpids(&buf, 16384*sizeof(int));
for (int i = 0; i != count; ++i) {
int pid = buf[i];
char path[MAXPATHLEN+1] = {0};
int ret = proc_pidinfo(pid, PROC_PIDPATHINFO, 0,
&path, sizeof(path));
if (ret < 0) {
printf("%d: error %s (%d)\n", pid, strerror(errno), errno);
} else {
printf("%d: %s\n", pid, path);
}
}
}
Obviously in real code you're going to want to allocate the buffer dynamically, return the values instead of just dumping them, get more than just the paths, etc. But this is enough to give you the basic idea. (When you go to get additional information, be aware that you if you ask for any struct, you will get an EPERM error unless you have rights to see every member of that struct. So, don't go asking for PROC_PIDTASKALLINFO if you only want PROC_PIDT_SHORTBSDINFO.
Anyway, since this API deals with BSD processes (and Mach tasks), not applications, it won't directly help you get at the NSBundle you want to provide Activity Monitor-style features.
There is no way to do this that's entirely correct, but you can probably get away with something like this:
NSString *path = processPath;
while (path && ![path isEqualTo:#"/"]) {
NSBundle *bundle = [NSBundle bundleWithPath:path];
if (bundle) {
if ([bundle executablePath != processPath]) return nil;
return bundle;
}
path = [path stringByDeletingLastPathComponent];
}
There are probably alternative ways to do this, each with different tradeoffs. For example, using -[NSWorkspace runningApplications], storing the results in a dictionary mapping the bundle executable path to the bundle, and using that to look up each process is simple, but it only seems to be useful for applications owned by the current user (and probably in the current session). On the other hand, enumerating all bundles on the system, or asking Spotlight, or similar would probably be too slow to do on the fly, but would go out of date if you cached them on first run.
Another option, in place of libproc, is to use libtop.
Unfortunately, Apple doesn't provide it. They do have a libtop implementation, which they use for their top tool, but it's actually embedded in the source to top and not available from outside. You can find the source (at the link above) and embed it into your program the same way top itself does.
Alternatively, both GNU and BSD process utilities have Mac ports (although knowing which name to use with Homebrew/MacPorts/Google search isn't always easy…), so you could build one of those and use it.
However, unless you're trying to write cross-platform software (or already know how to write this code for linux or FreeBSD or whatever), I think that just adds extra complexity.