Related
In advance apologize if the question seems somewhat broad or strange, I don't mean to offend anyone, but maybe someone can actually make a recommendation. I tried looking for the similar questions, but cold not.
Which are the better resources (books, blogs etc.) that can teach about optimizing code?
There is quite a few resources on making code more human-readable (Code Complete being number one choice probably). But what about making it run faster, more memory-efficient?
Of course there are lots of books on each particular language, but I wonder if there are some that cover the problems of memory / speed of operations and are somewhat language-independent?
Here are some links that might be helpful in general on the subject of memory optimizations
What Every Programmer Should Know About Memory by Ulrich Drepper
Herb Sutter: The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software
Slides: Herb Sutter: Machine Architecture (Things Your Programming Language Never Told You)
Video: Herb Sutter # NWCPP: Machine Architecture: Things Your Programming Language Never Told You
The microarchitecture of Intel, AMD and VIA CPUs
An optimization guide for assembly programmers and compiler makers, by Agner Fog
Read Structured Programming with go to Statements. While it's the source of the "premature optimisation is the source of all evil" quote that comes up the moment somebody wants to make anything faster or smaller - no matter how desperately important or late in the process they are - it's actually about the importance of making things efficient when you can.
Learn about time complexity, space complexity and the analysis of algorithms.
Come up with examples where you would want to sacrifice having worse space complexity for better time complexity, and vice versa.
Know the time and space complexities of the algorithms and data structures your languages and frameworks of choice offer, especially those you use most often.
Read the answers on this site on questions about creating a good hash code.
Study the approach HTTP took to having the advantage of caching, without the disadvantage of using stale data inappropriately. Consider how easy or difficult that is to apply to in-memory caches. Consider when you would say "screw it, I can live with being stale for the speed boost it gives me". Consider when you would say "screw it, I can live with being slow for the guarantee of freshness it gives me".
Learn how to multithread. Learn when it improves performance. Learn why it often doesn't or even makes things worse.
Look at a lot of Joe Duffy's blog where performance is a regular concern of his writing.
Learn how to process items as streams or iterations rather than building and rebuilding data-structures full of each item, each time. Learn when you're actually better off not doing that.
Know what things cost. You can't reasonably decide "I'll work so this is in the CPU cache rather than main-memory/main-memory rather than disk/disk rather than over a network" unless you've a good idea what actually causes each to be hit, and what the cost differences are. Worse, you can't dismiss something as premature optimisation if you don't know what they cost - not bothering to optimise something is often the best choice, but if you don't even consider it in passing you aren't "avoiding premature optimisation", you're muddling through and hoping it works.
Learn a bit about what optimisations are done for you by the script engine/jitter/compiler/etc you use. Learn how to work with them rather than against them. Learn not to re-do work it'll do for you anyway. In one or two cases, you may also be able to apply the same general principle to your work.
Search for cases on this site where something is dismissed as an implementation detail - yes, all of those are cases where the detail in question isn't the most important thing at the time, but all of those implementation details were chosen for a reason. Learn what they were. Learn the counter-arguments.
Edit (I'll keep adding a few more to this as I go):
Different books of course differ in the emphasis they put on efficiency concerns, but I remember Stroustrup's The C++ Programming Language as one where there were a good few times where he will explain a choice between a few different options as relating to efficiency, and also on how to not have decisions made for efficiency's sake impact on the usability of the classes "from the outside".
Which brings me to another point. Concentrate on the efficiency of the library code you reuse in different projects. You don't want to ever be thinking "maybe I should hand-roll a new one here to be more efficient", unless it's a very specialised case, you want to be confident that lots of work went into making that heavily used class efficient over a lot of case, and concentrate on identifying hot-spots.
As for specialised cases, some of the more obscure data structures are worth knowing for the cases they serve. For example, a DAWG is a very compact structure for storing strings with a lot of common prefixes and suffixes (which would be most words in most natural languages) where you just want to find those in the list that match a pattern. If you need a "payload" then a tree where each letter has a list of nodes for each subsequent letter (a generalisation of a DAWG but ending in that "payload" rather than the terminal node) has some but not all of the advantages. They also find the result in O(n) time where n is the length of the string sought.
How often will that come up? Not many. It came up for me once (a few times really, but they were variants of the same case), and as such it would not have been worth it for me to learn all there was to know about DAWGs until then. But I knew enough to know it was what I needed to research later, and it saved me gigabytes (really, from way too much for a machine with 16GB RAM to cope with, to less than 1.5GB). Going straight for a hand-rolled DAWG would totally be premature optimisation rather than putting the strings in a hashset, but flicking through the NIST datastructure site meant I could when it came up.
Consider: "Finding a string in a DAWG is O(n)" "Finding a string in a Hashset is O(1)" Both of these statements is true, but the speed of the two tends to be comparable. Why? Because the DAWG is O(n) in terms of the length of the string, and effectively O(1) in terms of the size of the DAWG. The Hashset is O(1) in terms of the size of the hashset, but working out the hash is typically O(n) in terms of the length of the string, and equality checks are also O(n) in terms of that length. Both statements were correct, but they were thinking about a different n! You always need to know what n means in any discussion of time and space complexity - most often it'll be the size of the structure, but not always.
Don't forget constant effects: O(n²) is the same as O(1) for sufficiently low values of n! Remember that the likes of O(n²) translates as n²*k + n * k₁ + k₂, with the assumption that k₁ & k₂ are low enough and k and the k of another algorithm or structure we are comparing of are close enough, that they don't really matter and it's only n² that we care about. This isn't true all the time, and we can sometimes find that k, k₁ or k₂ are high enough that we end up in trouble. It's also not true when n is going to be so small as to make the difference in the constant costs of different approaches matter. Of course normally when n is small we don't have a big efficiency concern, but what if we are doing m operations on structures averaging n in size, and m is large. If we are choosing between an O(1) and a O(n²) approach, we are choosing between an O(m) and O(n²m) approach overall. It still seems like a no-brainer in favour of the former, but with a low n it essentially becomes a choice between two different O(m) approaches, and the constant factors are much more important.
Learn about lock-free multi-threading. Or perhaps don't. Personally, I've two pieces of my own code I use professionally that use all but the simplest lock-free techniques. One is based on well-known approaches and I wouldn't bother now (it's .NET code first written for .NET2.0 and the .NET4.0 library supplies a class that does the same thing). The other I first wrote for fun, and only used after that just-for-fun period had given me something reliable (and it still gets beaten by something in the 4.0 library for a lot of cases, but not for some others that I care about). I would hate to have to write something like it with a deadline and a client in mind.
All that said, if you're coding out of interest, the challenges involved are interesting and it's an enjoyable thing to work with when you've the freedom to give up on a failed plan that you don't get when you're doing something for a paying client, and you'll certainly learn a lot about efficiency concerns generally. (Take a look at https://github.com/hackcraft/Ariadne if you want to see some of what I've done with this).
A Case Study
Actually, that contains a relatively good example of some of the above principles. Take a look at the method that's currently at line 511 at https://github.com/hackcraft/Ariadne/blob/master/Collections/ThreadSafeDictionary.cs (where I joke in the comments about it being flame-bait for people quoting Dijkstra. Let's use it as a case-study:
This method was first written to use recursion, because it's a naturally recursive problem - after doing the operation on the current table, if there's a "next" table we want to do the exact same operation on that, and so on until there's no further table.
Recursion is almost always slower than iteration, for a few different methods. Should we make all recursive calls iterative? No, it's often not worth it, and recursion is a wonderful way to write code that is clear about what it's doing. Here though I apply the principle above that since this is a library that might be called where performance is crucial, particular effort should be extended on it.
The decision to try to improve its speed being made, the next thing I did was make measurements. I don't depend on "I know that iteration is faster than recursion, so it must be faster when changed to avoid recursion". That's just not true - a poorly written iterative version may not be as good as a well-written recursive version.
The next question is, just how to re-write it. I've a tested method that I know works and I'm going to replace it with a different version. I don't want to replace it with a version that doesn't work, obviously, so how to re-write while taking the most advantage out of what's already there?
Well, I know about tail-call elimination; an optimisation normally done by compilers that changes the way the stack is managed so that recursive functions end up with properties closer to those of iterative (it's still recursive from the perspective of the source code, but it's iterative in terms of how the compiled code actually uses the stack).
This gives me two things to think about: 1. Maybe the compiler is already doing this, in which case my extra work isn't going to do anything to help. 2. If the compiler isn't already doing this, I can take the same basic approach manually.
That decision made, I replaced all of the points where the method called itself, with a change to the one parameter that would be different for that next call, and then go back to the beginning. I.e. instead of having:
CurrentMethod(param0.next, param1, param2, /*...*/);
We have:
param0 = param0.next;
goto startOfMethod;
That being done, I measure again. Running through the entire unit tests for the class is now consistently 13% faster than before. If it were closer I'd have tried more detail measurements, but a consistent 13% on runs that includes code that doesn't even call this method is something I'm pretty happy with. (It also tells me that the compiler wasn't doing the same optimisation, or I wouldn't have gained anything).
Then I clean up the method to make more changes that make sense with the new code. Most of them let me take out the goto because goto is indeed nasty (and there's other places the same optimisation was done that aren't as obvious because the goto was refactored entirely). In some, I left it in, because 13% is worth breaking the no-goto rule to my mind!
So the above gives an example of:
Deciding where to concentrate optimisation effort (based on how often it might be hit and my inability to predict all uses of the library)
Using knowledge of general costs (recursion costs more than iteration, most of the time).
Measuring rather than depending on assuming the above always applies.
Learning from what compilers do.
Understanding that because of that I may not gain anything - maybe the compiler already did it for me.
Avoiding optimisations leading to unreadable code (refactoring out most of the gotos the first pass introduced).
Some of these are matters of opinion and style (the decision to leave in some goto would not be without controversy), and it's certainly okay to disagree with my decisions, but knowledge of the points raised so far in this post would make it an informed disagreement, rather than a knee-jerk one.
In addition to the resources mentioned in other answers, Michael Abrash's Graphics Programming Black Book is a great read for learning about optimization. While the specifics are a bit dated in places, it is still a great resource for learning about how to approach optimization.
Any time you want to optimize code it is absolutely essential to measure, measure, measure. One of the best ways to learn about optimization is by doing - take some code you want to optimize, learn how to use a profiler to measure its performance and then make changes and measure the results.
In a sense NHibernate seems convenient because it leads to less typing, and then propably less errors.
I think NHibernate can be used in every size of application because it is really easy to use (especially with FluentNHibernate) and you it does much work for you like generating SQL
queries, mapping values to objects and so on. Even in typical small applications you need to put a great part of the whole effort on the data persistence layer, so why don't let NHibernate do the work for you?
Best Regards,
Oliver Hanappi
It is tough to judge what you are looking for based on the terseness of your question, as there is really a lot of nuance in an answer to this.
In many ways, as the others here say, it depends on your project and your knowledge of nHibernate. But it also depends on a lot of other factors as well...
If you think your small project might grow into a large project someday that could make a better argument for it as you then have a strong foundation on which to grow.
If your goal is learning nHibernate (or another ORM) then a small project may be the best place to get your feet wet and try it out. (Also try Linq2SQL, and other ORMs as well and go with what you find works the best for you.)
I personally use nHibernate for all of my projects large and small (where possible due to other constraints). But I've also been working with it for a while and a good base of code that I can reuse. So that factors into the time part of an answer. nHibernate has a pretty steep learning curve so if you need something done quick nHibernate may not be optimal.
I hope that helps, if you can refine your question and your goals a little more in your question it will assist the rest of us in getting you some better feedback and ideas to help you.
It depends what you mean by small, and how accomplished you are with Hibernate. I'd find that the extra overhead of getting set up would not make using it a good option on a little project personally. I'd say the same about other frameworks like Spring too, they are far more useful on larger projects with lots of developers.
It's not if you have a strong knowledge of NH, if you have some model generator, if you use some DBMS other than SQL Server.
I think that if you have a DB on SQL server, you're quin in a hurry, you are knowledgeable on LINQ, LINQ2SQL might be a good choice. Fast and RAD.
I agree with Oliver but would add the caveat that it is only easy to use once you know how (and that may take awhile). If you haven't gotten on top of the learning curve and need a simple app, NHib will slow you down; otherwise use it on anything that isn't completely trivial or a throw away! So your simple but effective app has one more sound piece of infrastructure as it scales and adapts to requirement changes.
HTH,
Berryl
Is it a good practice to follow optimization techniques during initial coding itself or should one concentrate purely on realization of functionality first?
If one concentrates purely on functionality during initial coding, then how easy or difficult is it to take care of optimization later on?
Optimise your design and architecture - don't lock yourself into a design which will never scale - but don't micro-optimise your implementation. In particular, don't sacrifice simplicity and readability for micro-optimised implementation... at least not without benchmarking your code (ideally your whole system) first.
Measurement really is the key point when it comes to performance. Bottlenecks are almost never where you expect them to be. There are loads of different ways of measuring; optimisation without any measurement is futile IMO.
Donald Knuth said:
We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil
It depends what you see as "optimization". Micro-optimization should not be done in early stages, and afterwards only if you have a valid reason to do so (e.g. profiler results or similar).
However, writing well-structured, clean code following best practices and common coding guidelines is a good habit, and once you're used to it, it doesn't take much more time than writing sloppy code. This kind of "optimization" (not the correct word for it, but some see it as such) should be done from the beginning.
See http://en.wikipedia.org/wiki/Program_optimization for quotes by Knuth.
If you believe that optimization might make your code harder to (a) get right in the first place, or (b) maintain in the long run, then it's probably best to get it right first. Having good development processes, such as Test Driven Development, can help you make optimisations later.
It's always better to have it work right and slow, than wrong and fast.
Rightly said by Donald Knuth "Premature optimization is the the root of all evil " , and it makes your coding speed slow. The best way to optimize is by visiting the codebase again and refactoring. This way you know which part of the code is often used or is a bottleneck and should be fine tuned.
Premature optimization is not a good thing.
And that goes especially for low level optimization. But at a higher level your design shouldn't lock out any future optimization.
For example.
The retrieval of collections should be hidden behind methods call, in the end you can always decide to cache the retrieval of collections or not.
After you have a stable application and(!) you have developed regression unit tests. You can profile the application and optimize the hotspots. And remember to after every optimalization step you should run your complete unit test set.
Is it a good practice to follow optimization techniques during initial coding itself or should one concentrate purely on realization of functionality first?
If you know performance is critical (or important), consider it in your design and write it correctly the first time. If you don't also consider this in your design and it is important, you are wasting time or "developing a proof of concept".
Part of this comes down to experience; If you know optimizations and your program's problem areas or have already implemented similar functionalities in the past, your experience will certainly help you create an implementation closer to the end result the first time. If you still need a proof of concept, you should not be writing the actual program until that's completed -- kick out some tests to determine what solution is appropriate for the problem, then implement it properly.
If one concentrates purely on functionality during initial coding, then how easy or difficult is it to take care of optimization later on?
Some fixes are quick, others deserve complete rewrites. The more that needs to change and adapt after the fact, the more time you waste re-testing and maintaining a poorly implemented program. The libraries that are easiest to maintain and sustain the demands are typically the ones which the engineer had an understanding of what design is ideal, and strived to meet that ideal during initial implementation.
Of course, that also assumes you favor a long-lived program!
Premature optimization is the the root of all evil
To elaborate more on this famous quote, doing optimization early has the disadvantage of distracting you from doing a good design. Also, programmers are notoriously bad in finding which parts of the code cause the more trouble, and so try hard to optimize things that aren't that important. You should always measure first to find out what needs to be optimized and this can only happen in later phases.
Actually I am making a major project in implementing compiler optimization techniques. I already know about the existing techniques, but I am confused what technique to choose and how to implement it.
G'day,
What area of optimization are you talking about?
Compiler optimizations such as:
loop optimizations
dataflow optimizations
static single assignment based optimizations
code generator optimizations
etc.
etc.
Or optimization in the performance of the compiler itself, i.e. the speed with which it works?
Assuming that you have a compiler to optimize, and if it wasn't written by you, look up the documentation to see what is missing. Otherwise, if it was written by you, you can start off with the simplest. The definition for the simplest will depend on the language your compiler consumes. Or am I missing something?
I think you may have over optimized your question . Are you trying to decide where to start or trying to decide if some optimizations are worth implementing and others are not? I would assume all of the existing techniques have a place and are useful depending on the code they come across. If you are deciding which one to do first, pick the one you can do and do it. Pick the low hanging fruit. Get a few wins in your back pocket before you tackle a tough one and stumble and get frustrated. I would assume the real trick is having all the optimizations there and working but coming up with a way to decide which ones produce something better for a particular program and which ones get in the way and make things worse.
IMHO, the thing to do is implement the simple, obvious optimizations and then let it rest. Certainly it is very interesting to try to do weird and wonderful optimizations to rectify things that the user could simply have coded a little better, but if you really want to try to clean up after poor coding or poor design, the user can always outrun you. This is my favorite example.
My favorite example of compiler-optimizations-gone-nuts is Fortran compilers, where they go to such lengths to scramble code to shave a few hypothetical cycles that the code is almost impossible to debug, and typically the program counter is in there less than 1% of the time, so the effort is wasted.
When applying the Single Responsibility Principle and looking at a class's reason to change, how do you determine whether that reason too change is too granular, or not granular enough?
I don't know that there's a good answer to this one other than "apply your judgement, based on your experience." Failing that, get help, which I guess is what you're doing here ;)
Seriously, though, if you find that you're creating a gazillion classes to do what seems like a simple job, then you're probably being too granular. If your classes all seem collossal, then you're probably being too coarse. Please pardon me if that's a statement of the obvious.
I think this is one of those fuzzy, no-hard-and-fast-rules cases that show us why we need human programmers. Just try something, seeking balance, and refactor if you find you're going too far in one direction or the other. And remember: if it's worth doing, it's worth doing badly.
I wouldn't be too worried about granularity initially. I will just go with separation of concern at a broader level initially. Basic point is that we should avoid over-engineering here. But just enough. I agree with Lucas here, that this first step will improve with experience.
As the requirements change, as I am starting to get the 'smells', as my understanding of the problem improves I would refactor the design by factoring out the separate concerns as they become obvious. Basically separation of concern shall also be evolutionary as with overall design.