I'm looking at adding scripting functionality to an existing codebase and am weighing up the pros/cons of various packages. Lua is probably the most obvious choice, but I was wondering if people have any other suggestions based on their experience.
Scripts will be triggered upon certain events and may stay resident for a period of time. For example upon startup a script may define several options which the program presents to the user as a number of buttons. Upon selecting one of these buttons the program will notify the script where further events may occur.
These are the only real requirements;
Must be a cross-platform library that is compilable from source
Scripts must be able to call registered code-side functions
Code must be able to call script-side functions
Be used within a C/C++ codebase.
Based on my own experience:
Python. IMHO this is a good choice. We have a pretty big code base with a lot of users and they like it a lot.
Ruby. There are some really nice apps such as Google Sketchup that use this. I wrote a Sketchup plugin and thought it was pretty nice.
Tcl. This is the old-school embeddable scripting language of choice, but it doesn't have a lot of momentum these days. It's high quality though, they use it on the Hubble Space Telescope!
Lua. I've only done baby stuff with it but IIRC it only has a floating point numeric type, so make sure that's not a problem for the data you will be working with.
We're lucky to be living in the golden age of scripting, so it's hard to make a bad choice if you choose from any of the popular ones.
I have played around a little bit with Spidermonkey. It seems like it would at least be worth a look at in your situation. I have heard good things about Lua as well. The big argument for using a javascript scripting language is that a lot of developers know it already and would probably be more comfortable from the get go, whereas Lua most likely would have a bit of a learning curve.
I'm not completely positive but I think that spidermonkey your 4 requirements.
I've used Python extensively for this purpose and have never regretted it.
Lua is has the most straight-forward C API for binding into a code base that I've ever used. In fact, I usually quickly roll bindings for it by hand. Whereas, you often wouldn't consider doing so without a generator like swig for others. Also, it's typically faster and more light weight than the alternatives, and coroutines are a very useful feature that few other languages provide.
AngelScript
lets you call standard C functions and C++ methods with no need for proxy functions. The application simply registers the functions, objects, and methods that the scripts should be able to work with and nothing more has to be done with your code. The same functions used by the application internally can also be used by the scripting engine, which eliminates the need to duplicate functionality.
For the script writer the scripting language follows the widely known syntax of C/C++ (with minor changes), but without the need to worry about pointers and memory leaks.
The original question described Tcl to a "T".
Tcl was designed from the beginning to be an embedded scripting language. It has evolved to be a first class dynamic language in its own right but still is used all over the world as an embeded language. It is available under the BSD license so it is just about as free as it gets. It also compiles on pretty much any moden platform, and many not-so-modern. And not only does it work on desktop systems, there are variations available for mobile platforms.
Tcl excels as a "glue" language, where you can write performance-intensive functions in C while still benefiting from the advantages of a scripting language for less performance critical parts of the application.
Tcl also comes with a first class GUI toolkit (Tk) that is arguably one of the easiest cross platform GUI toolkits available. It also interfaces very nicely with SQLite and other databases, and has had built-in support for unicode for quite some time.
If the scripting interface will be made available to your customers (as opposed to simply enabling your own engineers to work at the scripting level), Tcl is extremely easy to learn as there are a total of only 12 rules that govern the entire language (as of tcl 8.6). In fact, Tcl shines as a way to invent domain specific languages which is often how it is used as an end-user scripting solution.
There were some excellent suggestions already, but I just wanted to mention that Perl can also be called / can call to C/C++.
You probably could use any modern scripting / bytecode language.
If you're willing to put up with the growing pains of a new product, you could use the Parrot VM. Which has support for many, if not all of the languages listed on this page. Unfortunately it's not done yet, but that hasn't stopped some people from using it in a production environment.
I think most people are probably mentioning the scripting language that they are most familiar with. From my perspective, Tcl was designed specifically to interface with C, so your problem domain is tailor-made for the language. However, I'm sure Python, Perl, or Lua would be fine. You should probably choose the language that is most familiar to your current team, since that will reduce the learning time.
Related
I've been looking into building a basic game engine from the ground up and after making a list of features that are common to other engines, one of the bigger things is the fact that they have an embedded scripting language like lua or python.
My question is how is an embedded scripting language superior to just making a header file (or something of the like) that the user can include in a c++ file which gives them access to many of the functions and states. I'm sure there's a very good answer out there, I just haven't stumbled on it yet.
Also beyond why it's needed, what are languages like lua used for in things like game engines?
Lua is a far simpler language than C++, and all you need to edit it is a text editor. This puts the ability to script events and/or high level game logic in the hand of your designers and end users. Dynamic typing and garbage collecting allows them write very succinct code that focusses on game logic rather than all the systems-level housekeeping chores you get in a language like C++. It's also far easier to sandbox.
Lua is a popular choice because it's small, portable, hackable ANSI C code base; easy to embed, extend, and -- most importantly for game developers -- it has a minimal runtime footprint (one of the fastest interpreted languages). It's also a great combination of easy to learn/read/write syntax, but with powerful features like coroutines which can be very useful in games.
The reason for including a scripting language is to allow users to customize the behavior without having to recompile the code.
I'm not sure about what you are asking in the second part of the question. Are you asking what other languages are used, or are you asking what ways are languages like Lua used?
If you asked about what other languages are good for this, one such language is Tcl. Tcl was designed from the ground up to be an embedded scripting language, and is very mature and robust, and easily learned by non-technical people.
As for what scripting languages are good for ... configuration files is one way. By using a programming language rather than a text file with name/value pairs, it allows users to add logic to their start-up files. For example, maybe you allow users to assign different functions to keys on the keyboard; with a programming language they can add different functions for different computers. Or, if you're creating a game like a RPG, perhaps you can assign different keys for different character classes. If playing as a mage, F12 might be cast a spell, but if playing as a warrior f12 might be to do a finishing blow.
There are many ways to use scripting languages, and many different langages to choose from. It all boils down to allowing your users to customize the behavior of the game without having to recompile the code.
You might find this article by a Game developer useful in understanding why embedded languages are used.
http://www.grimrock.net/2012/07/25/making-of-grimrock-rapid-programming/
Another good reason, is unless you are sharing your source code with your game users and they are all C programmers, languages such as lua make it possible for users to extend the game, for example look at World of Warcraft.
There are many scripting language communities claiming that the language can be used for everything but in fact, nearly everybody uses it for one specific thing, e.g.: web development. If I take a look at Ruby, for example, they tell you its general-purpose but actually everybody is using it with rails for web development only..
Can you list me some uses of popular general-purpose scripting languages for the local PC? (except embedding) Are there any?
Is the fast development usually worth having to bring the whole interpreter with your program? Then there would be some language-dependent performance and stability problems too in most cases..
best regards,
lamas
I tend to use Python for most things that aren't compute bound, i.e. they aren't restricted by how many computations you do per second. Some of the things I've used Python for are:
General scripts to manipulate images etc. with the Python Imaging Library.
GUI frontends for command line applications using the pexpect module.
Mathematical modeling of microbial systems.
Bioinformatics.
Some web programming.
etc...
When the program/algorithm is compute bound, I use C together with Python and Ctypes. Does this fit your definition of general purpose? It's certainly useful for a wide variety of applications, but not suitable if the program needs to crunch numbers fast.
Stability: Python 2.5/2.6 is rock solid. Never had a crash that wasn't caused by self-stupidity.
Fast development: It's definitely worth it for me. For the most part, in the field where I work, programmer time is orders of magnitude more valuable than processor time. I'm quite happy to let a program run for hours if I can write it in a few days instead of a few weeks.
I often use PHP for things that I used to use bat files for. Much easier to write. Ironically, the deployment scripts to create installable materials for my web apps from the subversion sources are written in PHP.
Python is popular in the gaming community. EVE Online is written in python.
claiming that they can be used for everything but I often can't find any examples for that
You are basing your question on an incorrect assumption. Although, as pointed out, a Turing complete language will be able to compute what you require ... languages are 'viewed' by most as the sum of their most useful features and productive semantics.
The reality is:
Most scripting languages can do the same things, or support the most common things via libraries.
Some languages make a subset of operations more convenient, take Perl and regular expressions as an example
CPU time is cheap, as is RAM. Simple to understand code is the priority for most people.
The rise of the scripting languages is natural. Trying to assert any one language, approach or level of execution is good for a range of situations is usually fruitless.
What do you want?
What is the best language for that?
Is is fast enough or small enough? Usually the answer is yes
Imagine trying to use Python where you should be using Erlang, or C instead of Lisp because you thought all languages are equal. They aren't, even though, you can achieve the same things in a problem domain, in most languages/platforms with varying levels of ballache dependant on the task.
I often use ruby for what other people would create bash/sh files for. I find Ruby syntax intuitive for batch tasks along with a lot of other sorts of tasks(it's my goto language)
Perl is extremely popular for general scripting in unixes, such as there are package managers and websites and maintenance scripts written in perl.
Python is extremely popular for both web and application use.
VBA Is popular for being abused to write programs inside of Access, and also was once commonly used in ASP for websites (right?)
Nobody mentioned AppleScript!
Hahah, no seriously, Perl runs everywhere, is installed by default on (almost) any Unix-family OS (and is easy to get on Windows), and is extremely useful for gluing things together. And if you browse a bit at CPAN you'll see that it's extremely general-purpose. "Swiss army chainsaw" was intended as a slur but I think of it fondly. Performance is good too, though it hardly ever actually matters. Larry Wall's goal was "make easy things easy and hard things possible".
OK OK, so I'm a fanboy still, sigh.
I'm playing WoW for about 2 years and I was quite curious about Lua which is used to write addons. Since what I've read so far about Lua was "fast", "light" and "this is great", I was wondering how and when to use it.
What is the typical situation where you will need to embed a script language like Lua in a system ?
When you need end users to be able to define/change the system without requiring the system to rewritten. It's used in games to allow extensions or to allow the main game engine to remain unchanged, while allow content to be changed.
Embedded scripting languages work well for storing configuration information as well. Last I checked, the Mozilla family all use JavaScript for their config information.
Next up, they are great for developing plugins. You can create a custom API to expose to the plugin developers, and the plugin developers gain a lot of freedom from having an entire language to work with.
Another is when flat files aren't expressive enough. If you want to write data driven apps where behavior is parameterized, you'll get really tired of long strings of conditionals testing for config combinations. When this happens, you're better off writing the rules AND their evaluation into your config.
This topic gets some coverage in the book Pragramtic Programmer.
Lua is:
Lightweight
Easy to integrate, even in an asynchronized environment such as a game
Easy to learn for non-programmer staff such as integrators, designers and artists
Since games usually require all those qualities, Lua is mostly used there. Other sitation could be any application that needs some scripting functionality, but developers often opt for a little more heavy weight solution such as .Net or python.
In addition to the scripting and configurability cases mentioned, I would simply state that Lua+C (or Lua+C++) is a perfect match for any software development. It allows one to make an engine/usage interface where engine is done in C/C++ and the behaviour or customization done in Lua.
OS X Cocoa has Objective-C (C and Smalltalk amalgam, where language changes by the line). I find Lua+C similar, only the language changes by a source file, which to me is a better abstraction.
The reasons why you would not want to use Lua are also noteworthy. Because it hardly has a good debugger. Then again, people hardly seem to need one either. :)
a scripting language like Lua can also be used if you have to change code (with immediate effect) while the application is running. one may not see this in wow, because as far as i remember the code is loaded at the start (and not rechecked and reloaded while running).
but think of another example: webserver and scripting language - (thankfully) you can change your php code without having to recompile apache or restart apache.
steve yegge did that thing for his own mmorpg engine powering wyvern, using jython or rhino and javascript (can't remember). he wrote the core engine in java, but the program logic in python/javascript.
the effect of this is:
he doesn't have to restart the core engine when changing the scripts, because that would disconnect all the players
he can let others do the simpler programming like defining new items and monsters without exposing all the critical code to them
sandboxing: if an error happens inside the script, you may be able to handle it gracefully without endangering the surrounding application
Rapid development for application with real-time constraints. Computer games are one of these ;-)
It's a valid solution if you want to allow third parties to develop plug-ins or mods for your software.
You could implement an API in whatever language you are using, but a script language like LUA tends to be more simple and accessible for casual developers.
In addition to all the excellent reasons mentioned by others, Embedding Lua in C is very helpful when you need to manipulate text, work with files, or just need a higher level language. Lua has lots of nifty feature (Tables, functions are first class values, lots of other good stuff). Also, while lua isn't as fast as C or C++, it's pretty quick for an interpreted language.
For starters I should let you guys know what I'm trying to do. The project I'm working on has a requirement that requires a custom scripting system to be built. This will be used by non-programmers who are using the application and should be as close to natural language as possible. An example would be if the user needs to run a custom simulation and plot the output, the code they would write would need to look like
variable input1 is 10;
variable input2 is 20;
variable value1 is AVERAGE(input1, input2);
variable condition1 is true;
if condition1 then PLOT(value1);
Might not make a lot of sense, but its just an example. AVERAGE and PLOT are functions we'd like to define, they shouldn't be allowed to change them or really even see how they work. Is something like this possible with DLR? If not what other options would we have(start with ANTRL to define the grammar and then move on?)? In the future this may need to run using XBAP and WPF too, so this is also something we need to consider, but haven't seen much if anything on dlr & xbap. Thanks, and hopefully this all makes sense.
Lua is not an option as it is to different from what they are already accustomed to.
Ralf, its going to reactive, and to be honest the timeframe for when the results should get back to the user may be 1/100 of a second all the way up to 2 weeks or a month(very complex mathematical functions).
Basically they already have a system they purchased that does some of what they need, and included a custom scripting language that does what I mentioned above and they don't want to have to learn a new one, they basically just want us to copy it and add functionality. I think I'll just start with ANTRL and go from there.
Lua
it's small, fast, easy to embed, portable, extensible, and fun!
Lua is definitly the best choice for soft real-time system (like computer games).
See http://shootout.alioth.debian.org/ for detailed benchmarks.
However, last time I checked, Lua used a mark-and-sweep garbage collector which can lead to deadline-violation and non-deterministic jitter in real-time systems.
I believe that you could use theoretically use the DLR, but I'm unsure about support in an XBAP (partially trusted?) scenario.
If you host the DLR you would quickly be able to take advantage of IronRuby or IronPython scripting. You would want to look at these implementations when creating your own language implementation. If you post your question to the IronPython mailing list I'm sure you would get a better reply around the XBAP scenario, and some of the developers there created ToyScript.
What kind of real-time requirement are you trying to fulfill? Is the simulation a hard real-time simulation (some kind of hardware-in-the-loop simulation ==> deadline is less than 1/1000 second)?
Or do you want the scripting-system to be "reactive" to user-input ==> 1/10 should be sufficient.
I am no expert regarding MS DLR, but as far as I know, it does not support hard real-time systems. You may want to take a look at the real-time specification for Java (RTSJ)
Firstly I think that defining your own language is not the way to go.
Primarily because the biggest productivity gains you can get for programmers or non-programmers are the development tools. You (and 99.9% of the rest of us) are not going to write tools as good as what is out their.
Language design is hard.
Language support and documentation, also hard
I would recommend looking for a pre-built solution. If you could find a language that can lock down some functionality, that would be a good starting point. MatLab would be the first that comes to my mind.
Lastly, ditch the natural language part, BASIC, COBOL and YA-TDWTF-Lang all tried and failed at it.
Full disclosure: I work for a company that is developing a generalized domain specific language "system". It's targeted at data-in/text-out applications so it's not apropos and it's not yet to beta. The result is I'm somewhat knowledgeable and biased.
I'm a Java programmer, and I like my compiler, static analysis tools and unit testing frameworks as tools that help me quickly deliver robust and efficient code. The JRE is pretty much everywhere I would work, too.
Given that situation, I can't see a reason why I would ever choose to use shell scripting, vb scripting etc, no matter how small the task is if I wear one of my other hats like my cool black sysadmin fedora.
I don't wear the other hats too often, under what circumstances should I choose scripting over writing compiled code?
Whatever you think will be most efficient for you!
I had a co-worker who seemed to use a different language for every task; Perl for quick text processing, PHP for small internal web applications, .NET for our main product, cygwin for filesystem stuff. He preferred to use the technology which was most specific to the task at hand.
Personally, I find that context switching between technologies is painful. My day-to-day work is in .NET, so that's pretty much the terms I think in. For most tasks I find it more efficient to knock something up in C# using SnippetCompiler than I would to hack around in PowerShell or a scripting environment.
If you are comfortable with Java, and the JRE is everywhere you work, then I would say keep using it. There are, however, languages like perl and python that are particularly suited to quickly solving problems. I would suggest learning either perl or python, and then use your judgement on when to use it.
If I have a small problem that I'd like to solve quickly, I tend to use a scripting language. The code tax is smaller, and, for me at least, the result comes faster.
I would say where it makes sense. If it's going to take you longer to open up your IDE, compile the script, etc. than it would to edit a script file and be done with it than use script file. If you're not going to be changing the thing often and are quicker at Java coding then go that route :)
It is usually quicker to write scripts than compiled programmes. You don't have to worry so much about portability between different platforms and environments. A shell script will run pretty much every where on most platforms. Because you're a java developer and you mention that you have java everywhere you might look at groovy (http://groovy.codehaus.org/). It is a scripting language written in java with the ability to use java libraries.
The way I see it (others disagree) all your code needs to be maintainable. The smallest useful collection of code is that which a single person maintains. Even that benefits from the language and tools you mentioned.
However, there may obviously be tasks where specialised languages are more advantageous than a single general purpose language.
If you can write it quicker in Java, then go for it.
Just try and be aware of what the various scripting languages can do.
e.g. Don't make a full blown Java app when you can do the same with a bash one-liner.
Weigh the importance of the tool against popping open a text editor for a quick edit vs. opening IDE, recompiling, redeploying, etc.
Of course, the prime directive should be to "use whatever you're comfortable with." If Java is getting the job done right and on time, stick to it. But a lot of the scripting languages could save you some time because they're attuned to different problems. If you're using regular expressions, the scripting languages are a good fit. If you're dropping into shell commands, scripts are nice.
I tend to use Ruby scripts whenever I'm writing something that's small, because it's quick to write, easy to maintain, and (with Gems) easy to bolt on additional functionality without needed to use JARs or anything. Your milage will, of course, vary.
At the end of the day this is a question that only you can answer for yourself. Based on the fact that you said "I can't see a reason why I would ever choose to use shell scripting , ..." then it's probably the case that you should never choose it right now.
But if I were you I would pick a scripting language like python, ruby or perl and start trying to solve some of these small problems with this language. Over time you will start to get a feel for when it is more appropriate to write a quick script than build a full-blown solution.
I use scripting languages for writing programs which are not expected to be maintained beyond few executions. Most of these languages are light on boiler-plate syntax and do have a REPL. Both these features enable rapid prototyping.
Since you already know Java, you can try JVM languages like Groovy, JRuby, BeanShell etc. Scala has much lighter syntax than Java, has a REPL, is statically typed and runs on the JVM - you might give that a shot as well.