I´m trying to send a image through tcp to a server, firts getting the buffer from the camera and then converting to grayScale the buffer, finally I send the buffer to the server.
All is working fine, but the problem is that the image that the server receive it is not 100 % okay, it looks like there is some padding that I dind´t use at the conversion, all the images are more or less than the next.
I use the next code to get the image:
VImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
uint8_t * baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
the image is here http://s3.subirimagenes.com:81/imagen/previo/thump_6421684image001.png
The only padding you may get is per row of pixels — you should use something like:
/* ... */
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
for(interesting values of y)
{
uint8_t *pointerToThisLine = baseAddress + bytesPerRow*y;
}
Rather than assuming, one way or another, that one scanline ends somewhere in memory and then the next immediately starts.
That said, the top portion of your image is clearly correct and I've yet to see an instance where pitch wasn't equal to width*bytesPerPixel, so it'd be unlikely to be causing your problem in practice even if you haven't done that correctly.
Inspecting your image, it looks like the broken region contains copies of various fragments of the working region, so I don't think the problem is padding related — it's some sort of more obtuse memory management or transmission error. Have you checked that side of things?
Related
How to access the existing crop capabilities of vImage that are only documented for swift, but for objective-c?
https://developer.apple.com/documentation/accelerate/vimage/pixelbuffer/3951652-cropped?changes=_7_1&language=objc
just for linkage, i asked also on apple developer forum:
https://developer.apple.com/forums/thread/720851
In C, the computation is fairly straightforward, because the vImage_Buffer is just a pointer, height, width and rowBytes. It didn't exist for the first 20 years because it was assumed you could just do it trivially yourself. (Apple assumes familiarity with pointers in C based languages.) To be clear, you aren't actually cropping the image, just moving the pointer from the top left of the image to the top left of the sub rectangle and making the width and height smaller. The pixels stay where they are.
#include <Accelerate/Accelerate.h>
#include <CoreGraphics/CoreGraphics.h>
#define AdvancePtr( _ptr, _bytes) (__typeof__(_ptr))((uintptr_t)(_ptr) + (size_t)(_bytes))
static inline vImage_Buffer MyCrop( vImage_Buffer buf, CGRect where, size_t pixelBytes )
{
return (vImage_Buffer)
{
// irresponsibly assume where fits inside buf without checking
.data = AdvancePtr( buf.data, where.origin.y * buf.rowBytes + where.origin.x * pixelBytes ),
.height = (unsigned long) where.size.height, // irresponsibly assume where.size.height is an integer and not oversized
.width = (unsigned long) where.size.width, // irresponsibly assume where.size.width is an integer and not oversized
.rowBytes = buf.rowBytes
};
}
In Swift, there is less monkeying with raw pointers, so such methods may be deemed necessary.
Note that in certain cases with video content, wherein the "pixels" are actually glommed together in chunks, the calculation may be slightly different, and possibly the "pixel" may not be directly addressable at all. For example, if we had 422 content with YCbYCr 10-bit chunks (5 bytes/chunk), and you want to point to the second Y in the chunk, this wouldn't be possible because it would not be located at a byte addressable address. It would be spanned across a pair of bytes.
When it is calculable, the x part of the pointer movement would look like this:
(x_offset * bits_per_pixel) / 8 /*bits per byte*/
and we'd want to make sure that division was exact, without remainder. Most pixel formats have channels that are some integer multiple of a byte and don't suffer from this complication.
I try to add a B&W filter to the camera images of an ARSCNView, then render colored AR objects over it.
I'am almost there with the following code added to the beginning of - (void)renderer:(id<SCNSceneRenderer>)aRenderer updateAtTime:(NSTimeInterval)time
CVPixelBufferRef bg=self.sceneView.session.currentFrame.capturedImage;
if(bg){
char* k1 = CVPixelBufferGetBaseAddressOfPlane(bg, 1);
if(k1){
size_t x1 = CVPixelBufferGetWidthOfPlane(bg, 1);
size_t y1 = CVPixelBufferGetHeightOfPlane(bg, 1);
memset(k1, 128, x1*y1*2);
}
}
This works really fast on mobile, but here's the thing: sometimes a colored frame is displayed.
I've checked and my filtering code is executed but I assume it's too late, SceneKit's pipeline already processed camera input.
Calling the code earlier would help, but updateAtTime is the earliest point one can add custom frame by frame code.
Getting notifications on frame captures might help, but looks like the whole AVCapturesession is unaccessible.
The Metal ARKit example shows how to convert the camera image to RGB and that is the place where I would do filtering, but that shader is hidden when using SceneKit.
I've tried this possible answer but it's way too slow.
So how can I overcome the frame misses and convert the camera feed reliably to BW?
Here's the key for this problem:
session:didUpdateFrame:
Provides a newly captured camera image and accompanying AR information to the delegate.
So just moved CVPixelBufferRef manipulation, the image filtering code from
- (void)renderer:(id<SCNSceneRenderer>)aRenderer updateAtTime:(NSTimeInterval)time
to
- (void)session:(ARSession *)session didUpdateFrame:(ARFrame *)frame
Made sure to set self.sceneView.session.delegate = self to have this delegate called.
I have read (after running into the limitation myself) that for copying data from the host to a VK_IMAGE_TILING_OPTIMAL VkImage, you're better off using a VkBuffer rather than a VkImage for the staging image to avoid restrictions on mipmap and layer counts. (Here and Here)
So, when it came to implementing a glReadPixels-esque piece of functionality to read the results of a render-to-texture back to the host, I thought that reading to a staging VkBuffer with vkCmdCopyImageToBuffer instead of using a staging VkImage would be a good idea.
However, I haven't been able to get it to work yet, I'm seeing most of the intended image, but with rectangular blocks of the image in incorrect locations and even some bits duplicated.
There is a good chance that I've messed up my synchronization or layout transitions somewhere and I'll continue to investigate that possibility.
However, I couldn't figure out from the spec whether using vkCmdCopyImageToBuffer with an image source using VK_IMAGE_TILING_OPTIMAL is actually supposed to 'un-tile' the image, or whether I should actually expect to receive a garbled implementation-defined image layout if I attempt such a thing.
So my question is: Does vkCmdCopyImageToBuffer with a VK_IMAGE_TILING_OPTIMAL source image fill the buffer with linearly tiled data or optimally (implementation defined) tiled data?
Section 18.4 describes the layout of the data in the source/destination buffers, relative to the image being copied from/to. This is outlined in the description of the VkBufferImageCopy struct. There is no language in this section which would permit different behavior from tiled images.
The specification even has pseudo code for how copies work (this is for non-block compressed images):
rowLength = region->bufferRowLength;
if (rowLength == 0)
rowLength = region->imageExtent.width;
imageHeight = region->bufferImageHeight;
if (imageHeight == 0)
imageHeight = region->imageExtent.height;
texelSize = <texel size taken from the src/dstImage>;
address of (x,y,z) = region->bufferOffset + (((z * imageHeight) + y) * rowLength + x) * texelSize;
where x,y,z range from (0,0,0) to region->imageExtent.width,height,depth}.
The x,y,z part is the location of the pixel in question from the image. Since this location is not dependent on the tiling of the image (as evidenced by the lack of anything stating that it would be), buffer/image copies will work equally on both kinds of tiling.
Also, do note that this specification is shared between vkCmdCopyImageToBuffer and vkCmdCopyBufferToImage. As such, if a copy works one way, it by necessity must work the other.
For example, there are QR scanners which scan video stream in real time and get QR codes info.
I would like to check the light source from the video, if it is on or off, it is quite powerful so it is no problem.
I will probably take a video stream as input, maybe make images of it and analyze images or stream in real time for presence of light source (maybe number of pixels of certain color on the image?)
How do I approach this problem? Maybe there is some source of library?
It sounds like you are asking for information about several discreet steps. There are a multitude of ways to do each of them and if you get stuck on any individual step it would be a good idea to post a question about it individually.
1: Get video Frame
Like chaitanya.varanasi said, AVFoundation Framework is the best way of getting access to an video frame on IOS. If you want something less flexible and quicker try looking at open CV's video capture. The goal of this step is to get access to a pixel buffer from the camera. If you have trouble with this, ask about it specifically.
2: Put pixel buffer into OpenCV
This part is really easy. If you get it from openCV's video capture you are already done. If you get it from an AVFoundation you will need to put it into openCV like this
//Buffer is of type CVImageBufferRef, which is what AVFoundation should be giving you
//I assume it is BGRA or RGBA formatted, if it isn't, change CV_8UC4 to the appropriate format
CVPixelBufferLockBaseAddress( Buffer, 0 );
int bufferWidth = CVPixelBufferGetWidth(Buffer);
int bufferHeight = CVPixelBufferGetHeight(Buffer);
unsigned char *pixel = (unsigned char *)CVPixelBufferGetBaseAddress(Buffer);
cv::Mat image = cv::Mat(bufferHeight,bufferWidth,CV_8UC4,pixel); //put buffer in open cv, no memory copied
//Process image Here
//End processing
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );
note I am assuming you plan to do this in OpenCV since you used its tag. Also I assume you can get the OpenCV framework to link to your project. If that is an issue, ask a specific question about it.
3: Process Image
This part is by far the most open ended. All you have said about your problem is that you are trying to detect a strong light source. One very quick and easy way of doing that would be to detect the mean pixel value in a greyscale image. If you get the image in colour you can convert with cvtColor. Then just call Avg on it to get the mean value. Hopefully you can tell if the light is on by how that value fluctuates.
chaitanya.varanasi suggested another option, you should check it out too.
openCV is a very large library that can do a wide wide variety of things. Without knowing more about your problem I don't know what else to tell you.
Look at the AVFoundation Framework from Apple.
Hope it helps!
You can try this method: start by getting all images to an AVCaptureVideoDataOutput. From the method:captureOutput:didOutputSampleBuffer:fromConnection,you can sample/calculate every pixel. Source: answer
Also, you can take a look at this SO question where they check if a pixel is black. If its such a powerful light source, you can take the inverse of the pixel and then determine using a set threshold for black.
The above sample code only provides access to the pixel values stored in the buffer; you cannot run any other commands but those that change those values on a pixel-by-pixel basis:
for ( uint32_t y = 0; y < height; y++ )
{
for ( uint32_t x = 0; x < width; x++ )
{
bgraImage.at<cv::Vec<uint8_t,4> >(y,x)[1] = 0;
}
}
This—to use your example—will not work with the code you provided:
cv::Mat bgraImage = cv::Mat( (int)height, (int)extendedWidth, CV_8UC4, base );
cv::Mat grey = bgraImage.clone();
cv::cvtColor(grey, grey, 44);
recently I am implementing an crypto algorithm which uses 3DES. However, i found that the first 8 bytes of 4096 data block are always damaged. But it is sure that it can be decrypted correctly in java. Following is my code:
+ (void) DecryptBy3DES:(NSInputStream*)strmSrc Output:(NSOutputStream*)strmDest CryptoRef:(CCCryptorRef)tdesCrypto
{
size_t dataOutMoved;
uint8_t inputBuf[BlockSize];
uint8_t outputBuf[BlockSize];
CCCryptorStatus cryptStatus;
int iBytesRead = 0;
int iBuffUsed = 0;
while ( (iBytesRead = [strmSrc read:inputBuf maxLength:BlockSize]) > 0 )
{
cryptStatus = CCCryptorUpdate(tdesCrypto, &inputBuf, iBytesRead, &outputBuf, BlockSize, &dataOutMoved);
assert(cryptStatus==noErr);
[strmDest write:outputBuf maxLength:dataOutMoved];
}
CCCryptorReset(tdesCrypto, nil);
}
where BlockSize is 4096.
I reused the CCCryptoRef tdesCrypto to decrypt several blocks. The first block to be decrypted was correct, but the following blocks all had damaged bytes at the beginning. I also try to reset the CCCryptoRef, which seems in vain.
I am really confused. Anyone has the same problem?
Forget my previous answer, I deleted it. The reason that you get the "wrong bytes" in the buffer is that they are the last 8 bytes of plain text of the buffer you tried to decrypt before.
You must call CCCryptorFinal() right after the last call to CCCryptorUpdate(). This will remove the padding bytes before writing the last few bytes of plain text. Because the cipher internally does not know it the last block of the last buffer contains padding bytes, it can not write the data to the output buffer just yet.
Please do not destroy or reset the CCCryptor within your while loop. Simply add the call to CCCryptorFinal() right after, and don't forget to write the resulting output to stream as well. You may reset the CCCryptor after that.
I'm presuming (guessing) DESede with CBC mode and PKCS#5 padding here. See wikipedia to see what I am talking about.
Here is my CryptoRef:
CCCryptorCreateWithMode(kCCEncrypt, kCCModeCBC, kCCAlgorithm3DES, ccNoPadding, [abIV bytes], [abKey bytes], [abKey length], nil, 0, 0, kCCModeOptionCTR_BE, &cryptRef);
Since I use CBC mode and ccNoPadding, there is no need to call CCCryptorFinal(). Instead, when I finish one operation (i.e. finish encrypt/decrypt one file, etc.), I should call CCCryptorReset() to reset the CryptoRef's iv to initial state before next operation. Or the first block of data will be defect.
Thanks for comments and sorry for left this issue behind. I hope this could help people who encountered the same problem.