I'm using a Pharo 6.1 image to do some CSV file processing. Sometimes my image just becomes unresponsive — I have to terminate the Pharo VM from Task Manager.
Is it necessary to use somethings like:
ensure:[toStream close]
ensure:[fromStream close]
What's a simple but reliable approach to reading and especially writing files with Pharo?
The image is not really unresponsive but the UI thread for sure is when you do a long operation like reading a big CSV file.
The best would be to for a process for doing it.
Like in:
[ target doSomeLongThing ] fork.
You can view the process in the process browser from the world menu and terminate it there.
Now that is not really something one will think about when trying things out interactively in a playground.
What I do to alleviate this situation is twofold:
Put these things in a test. Why? Because tests have a maximum duration and if things are stuck, they will come back with a time exceeded notification from the test.
Start the Zinc REPL on a port, so that I can access the system from a web browser if the UI thread is kind of stuck and restart the UI thread.
I wish the interrupt key would work in all cases but well, it doesn't and is a solid annoyance.
I am of the opinion that images should be considered as discardable artifacts and CI job should be building them regularly (like daily) so that we have some recovery path.
This is not really cool as yes, using an image and being able to come back to it without having to rebuild stuff all the time when doing explorations shouldn't frustrate us with UI thread blockages. I hate that. I have an image that is stuck on restart from some Glamour problem, it is infuriating as it cannot be debugged or anything.
You can also use:
(FileLocation imageDirectory / 'somefile.csv') asFileReference readStreamDo: [ :stream |
"do something with the stream" ].
This will do the ensure: bit for you. Cleaner. Easier.
For CSV, also give a shot to NeoCSV as it will do the file handling for you as well.
HTH
Related
My Labview Program works like a charm, until I look at the Block Diagram. No changes are made. I do not Save. Just Ctrl+E and then Ctrl+R.
Now it does not work properly. Only a Restart of Labview fixes the problem.
My Program controls two Scanner arrays for Laser Cutting simultaneously. To force parallel working, I use the Error handler and loops that wait for a signal from the Scanner. But suddenly some loops run more often than they should.
What does majorly happen in Labview when I open the Block diagram that messes with my code?
Edit:
Its hard to tell what is happening without violating my non-disclosure agreement.
I'm controlling two independent mirror-Arrays for Laser Cutting. While one is running one Cutting-Job, the other is supposed to run the other Jobs. Just very fast. When the first is finished they meet at the same position and run the same geometry at the same slow speed. The jobs are provided as *.XML and stored as .net Objects. The device only runs the most recent job and overwrites it when getting a new one.
I can check if a job is still running. While this is true I run a while loop for the other jobs. Now this loop runs a few times too often and even ignores WAIT-blocks to a degree. Also it skips the part where it reads the XML job file, changes the speed part back to fast again and saves it. It only runs one time fast.
#Joe: No it does not. It only runs once well. afterwards it does not.
Youtube links
The way it is supposed to move
The wrong way
There is exactly one thing I can think of that changes solely by opening the block diagram.
When the block diagram opens, any commented-out or unreachable-code-compiler-eliminated sections of code will load their subVIs. If one of those commented out sections of code were somehow interfere with your running code, you might have an issue.
There are only two ways I know of for that to interfere... both of them are fairly improbable.
a) You have some sort of "check for all VIs in memory" or "check for all types in memory" that you're using as a plug-in system. When the commented-out sections load, that would change the VIs in memory. Such systems are not uncommon when parsing XML, so maybe.
b) You are using Run VI method for some dynamically invoked VI to execute as a top-level VI, but by loading the diagram, it discovers that it is a subVI of your current program. A VI cannot simultaneously be top-level and a subVI, so the call to Run VI returns an error.
That's it. I can't think of anything else. Both ideas seem unlikely, but given your claim and a lack of a block diagram, I figured I'd post it as a hypothesis.
In the improbable case someone has a similar problem. The problem was a xml file that was read during run time. Sometimes multiple instances tried to access it and this produced the error.
Quick point to check: are Debug and "retain data in wires" disabled? While it may not change the computations, but it may certainly change the timing of very tight loops, and that was one of the unexpected program behaviors, OP was referring to.
From what experience I have programming whenever a program has a problem it crashes, whether it is from an unhanded exception or a piece of code that should have been checked for errors, but was not and threw one. What would cause a program to completely freeze a system to the point of requiring a restart.
Edit: Thanks for the answers. As for the language and OS this question was inspired by me playing Fallout and the game freezing twice in an hour causing me to have to restart the xbox, so I am guessing c++.
A million different things. The most common that come to mind are:
Spawning too many threads or processes, which drowns the OS scheduler.
Gobbling too much RAM, which puts the memory manager into page-fault hell.
In a Dotnet/Java type environment its quite difficult to seize a system up, because the Runtime keeps you code at a distance from the OS.
Closer to the metal say C or C++, Assembly etc you have to play fair with the rest of the system - If you dont have it already grab a copy of Petzold and observe/experiment yourself with the amount of 'boilerplate' code to get a single Window running...
Even closer, down at the driver level all sorts of things can happen...
There are number of reasons, being internal or external that leads to deadlocked application, more general case is when something is being asked for by a program but is not given that leads to infinite waiting, the practical example to this is, a program writes some text to a file, but when it is about to open a file for writing, same file is opened by any other application, so the requesting app will wait (freeze in some cases if not coded properly) until it gets exclusive control of the file.
And a critical freeze that leads to restarting the system is when the file which is asked for is something which very important for the OS. However, you may not need to restart the system in order to get it back to normal, unless the program which was frozen is written in a language that produces native binary, i.e. C/C++ to be precise. So if application is written in a language which works with the concept of managed code, like any .NET language, it will not need a system restart to get things back to normal.
page faults, trying to access inaccessible data or memory(acces violation), incompatible data types etc.
I want to save my Pharo image every hour on the hour automatically.
How would you make this automatic within the image?
I've seen the Pier project do this.
But I'm not sure how they do it.
TIA
There is the Scheduler project on SqueakSource that looks like cron for Smalltalk. From the overview:
"Start a new task scheduler and keep it around"
scheduler := TaskScheduler new.
scheduler start.
"Let's save the image every hour"
scheduler
do: [Smalltalk snapshot: true andQuit: false]
every: 60 minutes.
You could combine that with the blocking code or OSProcess's saveImageInBackgroundNicely mentioned above and have a nice easy solution.
Result of a discussion on the mailing list, with some icing around to run it only hourly:
[[self blockUI.
self doUpdate.
SmalltalkImage current snapshot: true andQuit: false.
self unblockUI.
(Delay forDuration: (Duration hours: 1)) wait] repeat] fork
You can do it and it might work just fine.
But I wouldn't do it.
Not fot persistence in production.
Why?
Because images are like your session in your laptop. Saving your image is like putting your laptop to sleep: it persists everything.
And in the long run, some state will have some unexplainable shit that can complicate something and you will need to do a hard reboot.
It doesn't help to try to be perfectionist about it (or maybe it does but is certainly not economic). It will just happen and rebooting your laptop is the cheap solution to have fresh state. But that for your smalltalk app may not be that cheap.
A hard reboot in smalltalk will mean that you have to take a fresh image and load again all your code (it can be automated but experience tells that could be time consuming).
I'd like to have a small (not doing too damn much) daemon running on a little server, watching a directory for new files being added to it (and any directories in the main one), and calling another Clojure program to deal with that new file.
Ideally, each file would be added to a queue (a list represented by a ref in Clojure?) and the main process would take care of those files in the queue on a FIFO basis.
My question is: is having a JVM up running this little program all the time too much a resource hog? And do you have any suggestions as to how go about doing this?
Thank you very much!
EDIT: Another question I should ask: should I run this as its own instance (using less memory) and have it launch a new JVM when a file is seen, or have it on the same JVM the Clojure code that will process the file?
As long as it is running fine now and it has no memory leaks it should be fine.
From the daemon terminology I gather it is on a unix clone, and in this case best is to start it from an init script, or from the rc.local script. Unfortunately details differ from OS to OS to be more specific.
Limit the memry using -Xmx=64m or something to make sure it fails before taking down the rest of the services. Play a bit with the number to find the lowest reliable size.
Also, since clojures claim to fame is its ability to deal with concurrency it make a lot of sense to only run one JVM with all functionality running on it in multiple threads. The overhead of spawning new processes is already very big and if it is a JVM which needs to JIT and warm up its memory management, doubly so. On a resource constrained machine could pose a problem. and on a resource rich machine this is a waste.
I always found that the JVM is not made to quickly run something script like and exit again. It is really not made for that use case in my opinion
.
Whether this is possible I don't know, but it would mighty useful!
I have a process that fails periodically (running in Windows 2000). I then have just one chance to react to it before having to restart it and painfully wait for it to fail again. I didn't write the process so don't have the source to debug. The failure is seemingly random.
With a snapshot of the process I could repeatedly and quickly test reactions to the failure.
I had thought of running inside a VM but this isn't possible in this instance.
EDIT:
#Jon Cage asked:
When you say a snapshot, you mean capturing a process when it's about to fail (including memory, program state etc. etc.) ...and then replaying it's final few seconds repeatedly to see what effect it has on some other component?
This is exactly what I mean!
I think minidump is what you are looking for.
You can also used Userdump:
The User Mode Process Dumper
(userdump) dumps any running Win32
processes memory image (including
system processes such as csrss.exe,
winlogon.exe, services.exe, etc) on
the fly, without attaching a debugger,
or terminating target processes.
Generated dump file can be analyzed or
debugged by using the standard
debugging tools.
This article shows you how to use it.
My best bet is to start the process in a debugger (OllyDbg being my preferred tool).
The process will pause on an exception, and you can try to figure out what happened shortly before that.
This needs some understanding of assembler and does not allow to create a snapshot of the process for later analysis. You would need to write your own debugger for that - it should be theoretically possible.