The sequence of events that I'm trying to make happen in Meteor is:
On the client browser, upload a zip file and send it to the server
On the server, receive the zip file and hold it in a memory object
Unzip the memory object into individual objects representing the contents
Process the individual files one at a time
Return success/failure status to the client
I have steps 1 and 2 working, using EJSON to stringify the contents of the zip file on the client and again to convert it back to its original form on the server. The problem I'm encountering is when I try to unzip the object on the server. It seems that every unzip library available wants to operate directly on a file or stream, not on a memory object.
I suppose I could write the object to disk and read it back again, but that seems like an unnecessary step. Is there a library available to unzip a memory object? Alternatively, is there a way to create a stream directly from the object that I can then feed to the unzip routine?
Any advice would be greatly appreciated.
You could use the unzip module from npm. It accepts streaming input and allows you to process output without saving to disk.
It will take some work to wrap it to work with meteor. Your two options are the meteorhacks:npm package or upgrading to the Meteor 1.3 beta.
Related
I have a program that generates information from the contents of files, however, I believe it would be more efficient if I were able to do this as the files are being written; rather than having to then read the contents back after some delay, since I can simply generate the data as the file is writing to disk.
What method(s) are available for an application to hook into the file-write process, i.e- to process the data stream as it's being written to disk? Also, which of these (if any) are allowable for app store apps?
I've been considering using a Spotlight Importer, however this still involves reading the contents of a file after they've been written, in which case I'm relying on the file still being in the RAM cache to reduce disk access.
it is possible to use DotNetZip to create a zip from an accessed file (eg log file from another application) ?
so create a zip when the log file gets written through the other application
Hmm, well, yes, if you are willing to write some code.
One way to do it is to compress the file AFTER it has been written and closed.
You would need to have an app that runs with a filesystem watcher, and when it sees the log file being closed, it compresses that log file into a zip.
If you mean to imply, a distinct app that writes to a file and it automagically writes into a zip file, no I don't know of a simple way to do that. There is one possibility: if the 3rd party app accepts a System.IO.Stream in which to write the log entries. In that case, you can do that with DotNetZip. You can get a writeable stream from DotNetZip, into which the app writes content. It is compressed as it is written, and when the writing is complete, DotNetZip closes the zipfile. To use this, check the ZipFile.AddEntry() method that accepts a WriteDelegate. It's in the documentation.
I have a program which takes input from S3, generates a text file, and then sends it to the mapper class. I am unable to write the file to S3, from where the mapper can read it later. Now, I realize that we cannot write files to S3 directly, so I am trying to upload the text file created to S3 using copyFromLocalFile(). However, I get a null pointer exception in the following line:
fs.copyFromLocalFile(true, new Path(tgiPath), mapIP);
I am creating the text file in main function, so I am not sure where exactly it's being created. The only reason behind the null pointer exception, that I can think of is that the text file is not being written on the local disk. So my question is: How do I write files on the local disk? If I just specify the name of the file while creating it, where is it created and how do I access it?
Have a look at Jets3t
This seems to be exactly what you need.
Jets3t is awesome, but I am using Google's App Engine, and it doesn't work on there because of threading limitations.
I banged my head against the wall until I came up with a solution that worked on App Engine by combining a bunch of existing libraries: http://socialappdev.com/using-amazon-s3-with-google-app-engine-02-2011
On my site a user may upload a file (pic, zip, audio, video, whatever). He then may decide to replace it with a newer revision. This user may upload a file, make a post then decide to put up a new revision replacing the old (lets say its a large zip or tar.gz file). Theres a good chance people may be downloading it if he sent out an email or even im for the home user.
Problem. I need to replace the file and people may be downloading and it may be some minutes before it is deleted. I dont want my code to stall until i cant delete or check every second to see if its unused (especially bad if another user can start and he takes long creating a cycle).
How do i delete the file while users are downloading the file? i dont care if they stop i just care that the file can be replaced and new downloads are the new revision.
What about referencing the files indirectly?
A mapping script, maps a virtual file entry from your site to a real file . If the user wants to upload a new revision of his file you just update the mapping, not the real file.
You can install a daily task that scans all files and deletes all files without a mapping and without open connections.
lajuette's answer is right, the easiest solution is to work around the file locking altogether:
When a user uploads file foo.zip, internally store it as foo-v1.zip.
Create a mapping file somewhere (database, code, whatever) that maps foo.zip to foo-v1.zip.
Rather than exposing a direct link to the file, expose a link to a service that gets the file: mysite.com/Download?foo.zip or something. This service uses the mapping to determine which version of the file to send to the client.
When a new version is uploaded, create foo-v2.zip and update the mapping file.
It wouldn't be that hard to write a scheduled task that cleans up old, un-mapped files.
If your oppose to a database and If the filenames are in a fix format (such as user/id.ext) you could append the id with a revision number and enumerate the folder using a pattern (user/id-*) and use the latest revision.
I'm working with a GPS module that is transferring data to my mac over a serial RS232-to-USB interface. I've written a objC program that takes the raw data and converts it into meaningful information.
Using a program called goSerial, I'm able to log all incoming data into a text file. I have been able to make my program read the text file and process data line by line.
I would like this procedure to happen in real time i.e. as soon as the data is received, it gets logged into the text file and my program reads it. The first part of this happens automatically that is the text file is being constantly appended (when not open). Is it possible to monitor a text file for appended data and only read new lines? Also, will doing this affect the ability of new incoming data to be saved?
Thanks!!!
(Also, if anyone knows how I may send serial data directly to Xcode, please let me know!)
I'm not sure how the serial-to-USB affects things but traditionally, unix accesses serial devices using the Device-File Mechanism which treats the input from the device as a file to be read. You would use NSFileHandle to read the file from Cocoa/Foundation. You probably want to checkout the IORegistryExplorer app to see how your device shows up.
You can use a kqueue (perhaps with a wrapper such as UKKQueue) to watch the file for changes.
You should be able to create a unix domain socket, which you can then have your goSerial application open (as it looks like a normal file on the fs)
And then read from the other end, linewise in your application. This is probably the easiest way, or alternately have a look at the source of tail in GNU's coreutils, specifically it's -f function (although thinking more you'll probably want to look at how the FreeBSD implementation works, as I believe that the GNU version uses some linux specific callback)