We have an application that has some interesting memory usage issues. When it first opens, the program uses aroun 50-60MB of memory. This stays consistent on 32-bit machines. On 64-bit machines, however, re-activating the form in any way (clicking, dragging, alt-tabbing, etc.) adds around another 50MB to it's memory usage. It repeats this process several times before resetting back to around 45MB, at which point the cycle begins again.
I've done some research and a lot of people have said that VB in general has pretty poor garbage collection, which could be affecting the software in some way. However, I've yet to find a solution. There are no events fired when the application is activated (as shown by 32-bit usage) - the applications is merely sitting awaiting the user's actions.
At load, the system pulls some data into a tree view, but that's the only external connection, and it only re-fires the routine when the user makes a change to something and saves the change.
Has anyone else experienced anything this strange, and if so, does anyone know of what might fix it? It seems strange that it only occurs under x64 systems.
One of the best ways to start looking for what causes the leak is with a profiling tool.
http://www.jetbrains.com/profiler/
These guys have a 10 day free trial which is normally enough to figure out the area of the code that is causing your problems.
This doesn't sound like a problem at all. As mentioned in the comments, 64 bit programs will use more memory. The application doesn't use much memory, and it gets recovered after a while. VB.net doesn't have "pretty poor garbage collection", it uses the same collector as c#, the CLR's collector.
Related
Cheat Engine comes with a feature called speed hack which basically can slow down or increase speed of the game. Actually, not only games, if there is a software with clock ticking it can speed-en up that too. How does that work? I might imagine there is some internal clock on which these things run but not sure how these things happen on low level.
While this feature has worked on most of the games I tried, it has also failed on many, for eg, NFS Most wanted. Why? Is there any different mechanism on which these games run or it is just some anti-cheat?
Three years later, I think I know enough to answer my own question. :)
A computer program usually communicates with the kernel using predefined functions called system calls. Each OS has a different set of calls but often they do similar things like — allocating memory, reading and writing files, or handling processes. According to this page, there are around 480+ system calls in Windows NT kernel.
For any purpose that deals with the hardware, programs usually resort to system calls because that's what OS does best and one of these things happen to be knowing time. Computer games often need to render 60 frames / second and to make this happen they need to call the rendering function every 16.6ms. On Windows, "GetTickCount()" is usually used for this which returns number of milliseconds passed since the Windows has been up ("If no of milliseconds passed since the last tick count is more than 16ms, render a new frame else continue.").
SpeedHacking, in general, works by injecting code into the running process and hacking the timing functions to return sped-up / slowed-down "ticks" to modify the program's running speed.
Although, I can't be sure how exactly CE achieved this (the source code is pretty hard to understand) but another programmer pulled off a similar thing (video) on Linux. In the source code, you can see how the author modified a similar system call for Linux ("gettimeofday()") for this.
go gettimeofday_orig;
int val;
gettimeofday_orig=(go)dlsym(RTLD_NEXT,"gettimeofday");
if (!timezero)
{
timezero = new timeval;
val = gettimeofday_orig(timezero,tz);
(*tv) = (*timezero);
return val;
}
I am not sure how it's detected but I would be going with #Harold's idea that the game probably spots the DLL getting injected.
Cheat Engines Old Speedhack:
Runs the application in a very high priority thread
Uses timed sleeping to speed up the game
When a function is called, it will be given an emulated timer which is sped up
Cheat Engines New Speedhack:
When the Cheat Engine speed dll is injected into the program, it is modified to the speed you selected in the Cheat Engine panel
Sets a base reference of the current time
returned time = basetime+((currenttime-basetime)*speed
Detection:
Both Methods are easy to detect by sending a packet with the time every couple of seconds to be validated by a server
The game may be able to detect the dll being injected
Source:
http://wiki.cheatengine.org/index.php?title=Cheat_Engine:Internals#Speedhack
I think the reason why it does not work in some applications (mostly games) is that some games link the in-game clock to the frames per second. Therefore your game will slow down or crash if you try to speedhack it.
Source: http://hackerbot.net/tutorials/353-speed-up-hack-slow-down-cheat
Personally I have only encountered very few games that wouldnt react to the speedhack. Even if they are tied to the FPS, you can still speedhack it to some degree.
When reading this it said
Remember that all hardware, all firmware, and all software have faults
and introduce errors. Don’t trust anyone or anything. Have test
systems that bit flips and corrupts and ensure the production system
can operate through these faults – at scale, rare events are amazingly
common.
I don't understand. Is it possible to run any type of software if you flip bits? pointers will be incorrect and if you read for an address on a switch case (or even function call) you can start executing in no mans land. Heck, the stack could be overwritten. If your stack is gone how could you possibly recover. You can't, you have to terminate and restart again. Terminate+restart is hardly recover.
How do you 'test' a production system but forcing a part to terminate? Is this actually saying if the system is multiple CPU/systems unplug a system and see if it non unplugged system corrupts or crashes? Otherwise i don't understand what this is suppose to mean.
I think the main point in this is of increasing system robustness - see Jeff Atwood's Coding Horror post about the Netflix Chaos Monkey that randomly takes servers down. If you know the server is going down at some point (and most will) then you start planning for that circumstance. You add redundancy where needed, you harden the code to deal with situations like this, and you do it now instead of pushing it off which is easy to do when you're not directly feeling the pain.
A tangent off of this is that not every bit change will take down the system. It may just do things like corrupt data or inter-process communications. In that case, each component in your system needs to do its own error checking and not trust that it's always going to get good data from the other components.
By having a test system that causes these kinds of faults, you have a chance to fix them as appropriate before they become a huge problem with your customers.
I've got a kiosk app, which, essentially shows a bunch of slides with various bits of information on them. I initially began coding this over a year ago, when I was beginning with Objective-C and iOS development. I find that my code style is much cleaner now than what it was, and I'm much more experienced, so I've decided to rewrite from scratch.
I ran my app with the Allocations instrument to see what the memory usage was. Considering that this is a kiosk app, everything needs to run smoothly, without leaks. (Of course all apps need to run without leaks, but a kiosk app makes this an even more important goal.) I saw some interesting results, so I ran the old version of the code as well.
Running the older version of the code, I see a pretty much even run at about 1.15 megabytes of memory usage. Everything seems to be allocated and deallocated as necessary. In my new implementation, however, I'm seeing something a little different. Memory usage keeps jumping in little "plateaus", and then eventually seems to peak out at about 1.47 megabytes of usage. Here's what the new Allocations report looks like after running for over 10 hours:
I'm concerned for several reason.
The odd pattern in the beginning of the run.
Allocations seems to peak at 1.47 megabytes, but running it overnight shows that it actually will slowly use more and more memory over time. That can't be a good thing.
There are several notable differences between the old project and the new one.
The older one uses Plists as a backing store (I manually read and write to a plist file.) The new project uses Core Data.
The new project implements a library that is called on each "slide" that the old project didn't have. I'd be more concerned about this library, except I wrote it and I went through it to make sure I was releasing everything and only autoreleased wherever manual releases were impossible.
Both classes use a factory class to create the slides. In the old project, the factory class was a singleton. I thought that making it into a normal class would help with the memory issues, since the singleton was never released. (Hence it's properties were not being released.In the new project, the factory class is being released so I'm not sure why it's still taking up all that memory (if that's what's causing the problem.
The old project makes use of string constants in various places. The new code uses a massive enum for the same thing. (The new code in general uses more constants.)
What can I do to track down memory peaks? The memory is all being cleaned up by the application when it discards whatever it's using, but it doesn't seem to be discarding things until the app terminates.
I'd be grateful if anyone would help point me in the right direction.
Edit:
It looks like the peaking is being caused by calls to the KosherCocoa library. If anyone would mind taking a look at it and telling me what I'm doing wrong there as far as memory management goes, I'd really appreciate it.
What can I do to track down memory peaks? The memory is all being
cleaned up by the application when it discards whatever it's using,
but it doesn't seem to be discarding things.
This is a classic case of "abandoned objects" or "usage accretion". That is, you have an application that, as it runs, builds up an object graph in memory as a normal part of usage. The objects aren't leaked because they are still connected to the live object graph. More likely than not, the objects are a part of either some kind of a cache (a write-only cache, most often) or a mechanism involving historical state (the undo stack is a potential source for accretion).
To fix it, you need to make sure your object graph is pruned appropriately as your app runs. Caches should generally use a least-recently-used [LRU] pruning algorithm that limits the cache size. If a cache key ever goes invalid, that data should be pruned, too.
For historical information, pruning the history is critical. So is making sure that the historical data contains an absolutely minimal representation of that historical state.
Use Heapshot analysis -- it was created to help track down exactly these kinds of problems.
I wrote a detailed "How to" guide; When is a Leak not a Leak?
When things go badly awry in embedded systems I tend to write an error to a special log file in flash and then reboot (there's not much option if, say, you run out of memory).
I realize even that can go wrong, so I try to minimize it (by not allocating any memory during the final write, and boosting the write processes priority).
But that relies on someone retrieving the log file. Now I was considering sending a message over the intertubes to report the error before rebooting.
On second thoughts, of course, it would be better to send that message after reboot, but it did get me to thinking...
What sort of things ought I be doing if I discover an irrecoverable error, and how can I do them as safely as possible in a system which is in an unstable state?
One strategy is to use a section of RAM that is not initialised by during power-on/reboot. That can be used to store data that survives a reboot, and then when your app restarts, early on in the code it can check that memory and see if it contains any useful data. If it does, then write it to a log, or send it over a comms channel.
How to reserve a section of RAM that is non-initialised is platform-dependent, and depends if you're running a full-blown OS (Linux) that manages RAM initialisation or not. If you're on a small system where RAM initialisation is done by the C start-up code, then your compiler probably has a way to put data (a file-scope variable) in a different section (besides the usual e.g. .bss) which is not initialised by the C start-up code.
If the data is not initialised, then it will probably contain random data at power-up. To determine whether it contains random data or valid data, use a hash, e.g. CRC-32, to determine its validity. If your processor has a way to tell you if you're in a reboot vs a power-up reset, then you should also use that to decide that the data is invalid after a power-up.
There is no single answer to this. I would start with a Watchdog timer. This reboots the system if things go terribly awry.
Something else to consider - what is not in a log file is also important. If you have routine updates from various tasks/actions logged then you can learn from what is missing.
Finally, in the case that things go bad and you are still running: enter a critical section, turn off as much of the OS a possible, shut down peripherals, log as much state info as possible, then reboot!
The one thing you want to make sure you do is to not corrupt data that might legitimately be in flash, so if you try to write information in a crash situation you need to do so carefully and with the knowledge that the system might be an a very bad state so anything you do needs to be done in a way that doesn't make things worse.
Generally, when I detect a crash state I try to spit information out a serial port. A UART driver that's accessible from a crashed state is usually pretty simple - it just needs to be a simple polling driver that writes characters to the transmit data register when the busy bit is clear - a crash handler generally doesn't need to play nice with multitasking, so polling is fine. And it generally doesn't need to worry about incoming data; or at least not needing to worry about incoming data in a fashion that can't be handled by polling. In fact, a crash handler generally cannot expect that multitasking and interrupt handling will be working since the system is screwed up.
I try to have it write the register file, a portion of the stack and any important OS data structures (the current task control block or something) that might be available and interesting. A watchdog timer usually is responsible for resetting the system in this state, so the crash handler might not have the opportunity to write everything, so dump the most important stuff first (do not have the crash handler kick the watchdog - you don't want to have some bug mistakenly prevent the watchdog from resetting the system).
Of course this is most useful in a development setup, since when the device is released it might not have anything attached to the serial port. If you want to be able to capture these kinds of crash dumps after release, then they need to get written somewhere appropriate (like maybe a reserved section of flash - just make sure it's not part of the normal data/file system area unless you're sure it can't corrupt that data). Of course you'd need to have something examine that area at boot so it can be detected and sent somewhere useful or there's no point, unless you might get units back post-mortem and can hook them up to a debugging setup that can look at the data.
I think the most well known example of proper exception handling is a missile self-destruction. The exception was caused by arithmetic overflow in software. There obviously was a lot of tracing/recording media involved because the root cause is known. It was discovered debugged.
So, every embedded design must include 2 features: recording media like your log file and graceful halt, like disabling all timers/interrupts, shutting all ports and sitting in infinite loop or in case of a missile - self-destruction.
Writing messages to flash before reboot in embedded systems is often a bad idea. As you point out, no one is going to read the message, and if the problem is not transient you wear out the flash.
When the system is in an inconsistent state, there is almost nothing you can do reliably and the best thing to do is to restart the system as quickly as possible so that you can recover from transient failures (timing, special external events, etc.). In some systems I have written a trap handler that uses some reserved memory so that it can, set up the serial port and then emit a stack dump and register contents without requiring extra stack space or clobbering registers.
A simple restart with a dump like that is reasonable because if the problem is transient the restart will resolve the problem and you want to keep it simple and let the device continue. If the problem is not transient you are not going to make forward progress anyway and someone can come along and connect a diagnostic device.
Very interesting paper on failures and recovery: WHY DO COMPUTERS STOP AND WHAT CAN BE DONE ABOUT IT?
For a very simple system, do you have a pin you can wiggle? For example, when you start up configure it to have high output, if things go way south (i.e. watchdog reset pending) then set it to low.
Have you ever considered using a garbage collector ?
And I'm not joking.
If you do dynamic allocation at runtime in embedded systems,
why not reserve a mark buffer and mark and sweep when the excrement hits the rotating air blower.
You've probably got the malloc (or whatever) implementation's source, right ?
If you don't have library sources for your embedded system forget I ever suggested it, but tell the rest of us what equipment it is in so we can avoid ever using it. Yikes (how do you debug without library sources?).
If you're system is already dead.... who cares how long it takes. It obviously isn't critical that it be running this instant;
if it was you couldn't risk "dieing" like this anyway ?
You may know a lot of programs, e.g some password cracking programs, we can stop them while they're running, and when we run the program again (with or without entering a same input), they will be able to continue from where they have left. I wonder what kind of technique those programs are using?
[Edit] I am writing a program mainly based on recursion functions. Within my knowledge, I think it is incredibly difficult to save such states in my program. Is there any technique, somehow, saves the stack contents, function calls, and data involved in my program, and then when it is restarted, it can run as if it hasn't been stopped? This is just some concepts I got in my mind, so please forgive me if it doesn't make sense...
It's going to be different for every program. For something as simple as, say, a brute force password cracker all that would really need to be saved was the last password tried. For other apps you may need to store several data points, but that's really all there is too it: saving and loading the minimum amount of information needed to reconstruct where you were.
Another common technique is to save an image of the entire program state. If you've ever played with a game console emulator with the ability to save state, this is how they do it. A similar technique exists in Python with pickling. If the environment is stable enough (ie: no varying pointers) you simply copy the entire apps memory state into a binary file. When you want to resume, you copy it back into memory and begin running again. This gives you near perfect state recovery, but whether or not it's at all possible is highly environment/language dependent. (For example: most C++ apps couldn't do this without help from the OS or if they were built VERY carefully with this in mind.)
Use Persistence.
Persistence is a mechanism through which the life of an object is beyond programs execution lifetime.
Store the state of the objects involved in the process on the local hard drive using serialization.
Implement Persistent Objects with Java Serialization
To achieve this, you need to continually save state (i.e. where you are in your calculation). This way, if you interrupt the probram, when it restarts, it will know it is in the middle of calculation, and where it was in that calculation.
You also probably want to have your main calculation in a separate thread from your user interface - this way you can respond to "close / interrupt" requests from your user interface and handle them appropriately by stopping / pausing the thread.
For linux, there is a project named CRIU, which supports process-level save and resume. It is quite like hibernation and resuming of the OS, but the granularity is broken down to processes. It also supports container technologies, specifically Docker. Refer to http://criu.org/ for more information.