I have an plugin written in Objective-C which incorporates a CoreML model. The plugin and ML model compile and run fine locally. If I email or transfer the plugin model and coreml model via Box, my plugin crashes and throws a damaged error. I can get the plugin to function by removing extended attributes in terminal: xattr -cr me/myplugin.plugin but the ML section of code still fails.
If I monitor in XCode, I notice the following when the coreml model fails:
[coreml] Input feature input_layer required but not passed to neural network.
[coreml] Failure verifying inputs.
Is there some signature or attached attribute that would lead to this issue when transferring via email/box?
Is there some signature or attached attribute that would lead to this
issue when transferring via email/box?
Since you have access to both versions of each file (before emailing / transferring via box, and the after transferring).
Go to both versions of each file and do the following:
ls -la
If it has extended attributes there will be an # symbol. For example:
drwxr-xr-x# 254 hoakley staff 8636 24 Jul 18:39 miscDocs
If the versions after transfer do not have an # symbol, then they do not have extended attributes.
Then for each file (both versions) do:
xattr -l filepath
This will display the extended attributes of each file.
You should compare the attributes of both versions of each file and see the difference. This should answer your question. If there is no difference, then no extended attribute has been added or removed.
Read : https://eclecticlight.co/2017/08/14/show-me-your-metadata-extended-attributes-in-macos-sierra/
Related
Is there a way to invoke bazel help such that it outputs a machine readable format. I would like to parse all the flags that are available in Bazel and make them automatically available in ibazel so that I don't have to manually sync them every time a new bazel release comes out with different flags.
There used to be a --helpxml argument that printed things out as XML, but it seems the command line argument parser has changed and you can no longer use that. I presume there is still some way to get this, since the docs are being generated with up to date command line info. Unfortunately the "edit this page" button on the docs site 404s and I can't figure out its origin.
The docs for the flags are generated via this genrule, which essentially runs bazel help everything-as-html, the source of which is here.
There are a few other options listed in that case statement, one of which being flags-as-proto which emits the flags as a base64 encoded version of the BazelFlagsProto.
Potentially ibazel could read this in, load in the proto and pull out the data from there.
As described in the documentation, using the version_labels field, you can specify a label to a model version in order to handle canary deployments.
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/serving_config.md#assigning-string-labels-to-model-versions-to-simplify-canary-and-rollback
For example, you can have model 43 labeled as stable and model 44 labeled as canary.
That feature sounds really neat, but I did not find in the doc how to adapt my POST request to specify the label I want to use.
Until now, I was using something of the sort:
curl -d '{"instances": <<my input data>>}' -X POST http://localhost:8501/v1/models/<<my model name>>:predict
Any idea ?
Update:
Based on comments on this GitHub Issue, #misterpeddy states that, as of August 14th 2019:
Re: not being able to access the version using labels via HTTP - this is something that's not possible today (AFAIR) - only through the grpc interface can you declare labels :(
To the best of my knowledge, this feature is yet to be implemented.
Original Answer:
It looks like the current implementation of the HTTP API Handler expects the version to be numeric.
You can see the regular expression that attempts to parse the URL here.
prediction_api_regex_(
R"((?i)/v1/models/([^/:]+)(?:/versions/(\d+))?:(classify|regress|predict))")
The \d defines an expectation for a numeric version indicator rather than a text label.
I've opened a corresponding TensorFlow Serving issue here.
The REST API for TensorFlow Serving is defined here: https://www.tensorflow.org/tfx/serving/api_rest#url_4
For the predict method it would be:
http://host:port/v1/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]:predict
where ${MODEL_VERSION} would be stable or canary
Disclaimer: I posted this originally on Code Review but as the code does not currently work I was suggested to post at SO instead.
I have converted the official AWS SDK iOS (v 2.5.0) framework (focusing on S3) to macOS and everything is working as expected with uploads and downloads. However, I wanted to also enable Transfer Acceleration for S3 using AWSTransferManager. I know that you can enable Transfer Acceleration (TA) using the AWSTransferUtility, but that utility uses pre-signed requests that are valid for only 50 minutes (useful for iOS but not macOS). I would like to be able to transfer files that are large and can take hours even when using Transfer Acceleration.
I have edited the original code from AWS to enable TA for AWSTransferManager, however, I still can not get this to work properly as the final signing of the upload/download request fails. The error message is:
Message=The request signature we calculated does not match the signature you provided. Check your key and signing method.}]
For the most part, I have edited the files AWSSignature, AWSS3TransferManager, and AWSService (AWSServiceConfiguration). I think that the signing error occurs because I am editing a path, or URL without correctly code signing the change (probably in AWSSignature.m). As I am uncertain as to where my code breaks, I have created a repository with all of the AWS SKD macOS files required to compile the framework including a unit test. If I run the test initializing the call to AWSServiceConfiguration with:
AWSServiceConfiguration *configuration = [[AWSServiceConfiguration alloc]initWithRegion:[region aws_regionTypeValue] credentialsProvider:credentialsProvider
accelerateModeEnabled:#(NO)
bucketName:self.testBucketName];
Then everything works as expected and the test file uploads and downloads correctly. However, if I try to turn transfer acceleration on (I have already made sure that my bucket has acceleration enabled), then it fails with the code signing error above.
AWSServiceConfiguration *configuration = [[AWSServiceConfiguration alloc]initWithRegion:[region aws_regionTypeValue] credentialsProvider:credentialsProvider
accelerateModeEnabled:#(YES)
bucketName:self.testBucketName];
In fact it seems like the test script is trying to upload a file much larger than what the testfile (3 Mb) really is. I assume my error is related to the signing of the URL body in some way as the size of the file seems wrong.
I know that finding this error requires a bigger effort than what is usually expected at SO and it is not something many will throw themselves at (but hopefully some) as it involves very complex code and it is time-consuming. However, I do believe that if we could make this work then many can find the framework for AWS SKD for macOS + Transfer Acceleration very useful.
I hope you will take a look at try to identify what may be my problem breaking the code signing.
All code for the framework+test example is available here: https://github.com/trondkr/aws-sdk-macos-TA. To run the unit test you need to provide the secret key and access id to your AWS S3 and the name of a bucket that has transfer acceleration enabled.
Thanks. Cheers, Trond
I am going to check loading and memory path of process to find malicious processes. for example if csrss.exe is executed from other path than Windows\System32 would be considered malicious. But the result of Volatility for common process such as csrss.exe is as follows:
loading path : \??\C:\WINDOWS\system32\csrss.exe
mapped path : \WINDOWS\system32\csrss.exe
or for smss.exe I have
loading path : \SystemRoot\System32\smss.exe
mapped path : \WINDOWS\system32\smss.exe
So are these two paths equal in these two examples or not ? I.e. is \??\C:\WINDOWS == \WINDOWS
or \SystemRoot\System32 == \WINDOWS\system32
No, a ??\C:\... without the leading \ is not valid, to the best of my knowledge. Perhaps some libraries will accept this as input, however, and convert it appropriately. Or perhaps you missed some bits from the paths you copied and pasted?
Either way we'll stick to the (NT flavored) Windows behavior.
However, \??\C:\... definitely is. You can find a comprehensive discussion here and the official documentation here. Some of the aspects we need to know are also explained here.
Some implementation details
Quoting from the Google Project Zero page above:
There are 7 types of path that the Win32 API distinguishes between,
and potentially does different things with. NTDLL has a function,
RtlDetermineDosPathNameType_U, which, given a Unicode string will
return you the path type. We’ll go through each one of these types in
the next section. The following prototype can be used to call this
function:
enum RTL_PATH_TYPE {
RtlPathTypeUnknown,
RtlPathTypeUncAbsolute,
RtlPathTypeDriveAbsolute,
RtlPathTypeDriveRelative,
RtlPathTypeRooted,
RtlPathTypeRelative,
RtlPathTypeLocalDevice,
RtlPathTypeRootLocalDevice
};
RTL_PATH_TYPE NTAPI RtlDetermineDosPathNameType_U(_In_ PCWSTR Path);
To follow along, download WinObj from Sysinternals/Microsoft and start it elevated (right-click + "Run as Administrator").
Potentially valid paths you may see
You have a number of path representations (as discussed in-depth in the linked articles). Let's consider the path C:\Windows\System32\csrss.exe you gave:
Win32 path: C:\Windows\System32\csrss.exe (\Windows\System32\csrss.exe would be a variation that depends on the current directory at the time the path is processed, if it is somewhere on C: they should be identical)
NT native path:
\??\C:\Windows\System32\csrss.exe
\GLOBAL??\C:\Windows\System32\csrss.exe
\SystemRoot\System32\csrss.exe
Local device path: \\.\C:\Windows\System32\csrss.exe
Root local device path: \\?\C:\Windows\System32\csrss.exe
DOS paths: this is relevant for all paths whose individual path segments are longer than the classic 8.3 DOS path name segments. If configured, a 8.3 DOS name will be generated (usually something in the form of C:\LongPathName -> C:\LONGPA~1) ... this may be important for forensic purposes (you quote volatility), but is mostly legacy behavior.
UNC paths also exist and in fact a local device path (\\.\...) looks like a UNC path with server name . which could even be a thing, considering . is the current directory further down in paths, so why not the current machine up top? Either way, go read the articles if you want to know about those in more detail.
Starting at the top, drilling down
C:\Windows\System32\csrss.exe when passed to an API like CreateFile will be converted to \??\C:\Windows\System32\csrss.exe which the object manager then takes in for processing.
If a relative path was given, there will be additionally processing to convert to a full path, basically.
\\.\C:\Windows\System32\csrss.exe when passed to an API like CreateFile will be "converted" to \??\C:\Windows\System32\csrss.exe after some processing. Just like without the \\.\ prefix some additional processing happens before this path is passed down to the object manager.
\\?\C:\Windows\System32\csrss.exe when passed to an API like CreateFile will be "converted" to \??\C:\Windows\System32\csrss.exe. No further processing happens at the Win32 level. This is basically an escape route straight to the object manager (and in the Fun facts section I'll share a related fun fact).
The object manager view
Receiving a path like \??\C:\Windows\System32\csrss.exe the object manager attempts to find \?? first.
\?? used to be an actual item, these days it is a "view" of \GLOBAL?? overlaid with your session-specific DosDevices.
An example of session-specific DosDevices would be mapped network shares.
For a drive letter F: mapped to \\fileserver\Share
Under the hood you will have a session-specific DosDevices object directory like \Sessions\0\DosDevices\00000000-00261e8a which would contain a symbolic link named F:, pointing to \Device\LanmanRedirector\;F:0000000000261e8a\fileserver\Share ... or similar.
So in essence when looking up names in \?? the object manager will look in your session specific DosDevices, followed by \GLOBAL?? (or was it the other way around? 🤔)
Another example would be VeraCrypt mounted volumes. For example your drive letter T: may map to \Device\VeraCryptVolumeT
Having figured out that C: is \GLOBAL??\C: in our case, the object manager checks what C: is. On my Windows 10 it figures out that this is a symbolic link pointing to \Device\HarddiskVolume3.
Having found the device (and implicitly the driver associated with it) the object manager now will pass the remainder of the path (\Windows\System32\csrss.exe) to the device \Device\HarddiskVolume3, whose driver will hopefully do the right thing, create a file object and so on ...
And now that you know this you can follow along on your system for arbitrary paths.
What about \SystemRoot?
Well, on modern Windows versions the behavior differs a little from how it used to be, but \SystemRoot is a symbolic link resolving to the Windows directory on all NT-based Windows versions I am aware of. The Win32 subsystem parrots this in the %SystemRoot% environment variable (albeit in Win32 path notation).
On my Windows 10 \SystemRoot is a symbolic link to \Device\BootDevice\windows and so the path to a driver (which is the typical use case in the registry) can be expressed much more easily for early boot drivers (less processing needed by the object manager).
As a side note: on my system - as you'd expect - \Device\BootDevice maps to \Device\HarddiskVolume3.
Fun facts
The DosDevices object directory cited in the Google Project Zero article is no longer a thing. These days it's a symbolic link to \?? (not \GLOBAL??).
The subst command line utility allows you to create symbolic links inside your session-specific DosDevices object directory to map some directory path to a drive letter (under the hood uses DefineDosDevice()).
Files are unnamed objects, that is they have no name that could be directly processed by the NT object manager. Instead the remainder of a file name - after reparsing etc - will be passed to the driver responsible for the device object representing the file system volume (i.e. partition). That driver then takes care of making sense of the name.
\REGISTRY\MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem (while this doesn't name a file, it is a perfectly valid native path to what you would see as HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
That escape routes named \??\ and \\?\
I mentioned before that the prefix \??\ is an escape route. Consider a file/folder path with a trailing dot, something you will not be able to create or access ordinarily from the Win32 subsystem. Similarly the reserved DOS device names (e.g. NUL, COM1 etc.) cannot be created or manipulated ordinarily.
Little experiment ...
start cmd.exe (Win+R)
go to your desktop folder: cd /d "%USERPROFILE%\Desktop"
Now create a new file test.txt. (note the trailing dot): echo . > \??\%CD%\test.txt. (the %CD% refers to the current directory)
Use Windows Explorer to navigate to your desktop folder. Try to open the file test.txt. by double-clicking. Pick Notepad when it asks you what program to use to open it.
You should see:
... please note the path (without trailing dot).
Now try to delete the file from Windows Explorer. It won't work.
NB: Some alternative file managers like Far or Speed Commander won't have trouble with this!
Now switch back to the command line window and use del /f \\?\%CD%\test.txt.
Did you notice something? I sneakily changed from \??\ to \\?\ because apparently the implementation of del doesn't like the former.
Depending on the implementation of your specific program it can vary whether \??\ or \\?\ is successful.
Another experiment would be to create reserved names like NUL or COM1 or tinker with WSL2's per-directory case-sensitivity.
Note
Win32 may sound outdated, but the Win32 subsystem and the respective "Windows personality" implemented through csrss.exe and its subsystem DLLs (kernel32.dll and friends) was called just that. Besides, if you're picky it may bother you particularly to find out that all the native 64-bit binaries on a modern x64 Windows reside in %SystemRoot%\System32 (<- yes, that's a 32 right there 😁). Enjoy!
Further reading
In addition to the articles from the top, I can recommend these books:
Obviously the Windows Internals book series.
Windows System Programming by Pavel Yosifovich also has some details
I posted in the Yii forum with no luck. I am hoping someone in this community got this extension working because I really need it!
I used the yiic docs check command, and all the models seem to finally pass the check. So, I proceeded to executing this command:
yiic docs C:\path to my site\protected\models
And at first this appears onscreen:
Building.. : MyApplication Class Reference
Version... : 1.1.7
Source URL:
And then this error pops up:
Building model...
PHP Error[2]:include<GxActiveRecord.php>: failed to open stream: No such file or directory in file C:\path-to-my-app\yii\framework\YiiBase.php at line 396
Where, at that line, I have the following code:
include($className.'.php');
I don't understand why it is looking for a class I do not have!! I found it here, but I don't know in what specific folder that class should reside, or better yet why I should have to download it separately when it's not specified in the instructions. Thanks.
Looks like you've generated models from giix which have GxActiveRecord as a base class and the doc parser does not find this base class, usually you've defined this as an import in your application config, so this may be an expected behavior.
You could try importing something like 'ext.giix.components.GxActiveRecord' in every one of your model classes or copy GxActiveRecord into your models directory just for generating the docs.