We have an application which at certain times needs to execute 'tasks' in order to do some work. These tasks currently are just commands (read: dodgy VB scripts) being fired off and then monitoring the process for completion or aborting if it 'times out', however we don't have enough control over the execution.
I would like to integrate a scripting engine or scripting host to enable us to execute the scripts within our application (in another appdomain if possible) and maintain some control over them. I'd envisage that we have scripts which follow an interface similar to:
void Initialize();
void Destroy();
void Execute();
void Abort();
int GetProgress();
Of course, the above is probably not what would be in the code, however you get the idea... Essentially from our application we could call Initialize, then Execute, poll GetProgress until the return value is 100 or whatever, then at the end either call Destroy or Abort then Destroy if we need to cancel or it goes past the timeout period.
The main thing that the scripts would need to be able to do is AD queries, and interact with the Shell (e.g. Map a network drive, or Disconnect one).
I've looked into a few options to do this - one which I have implemented a prototype of uses Boo as the language and essentially loads the script on the fly to compile and execute as an 'assembly' in memory. I like the way this works but I'm not sure how well it would be accepted not being a 'well known' language etc.
Other options I'm looking at are Python (IronPython), Ruby (IronRuby), PowerShell, and maybe VB, but I'm not sure about the ease of implementation with VB.
Any opinions, comments, suggestions or even resources which might point me in the right direction?
Lua is designed exactly to be embedded in applications. it doesn't hurt that it's a really neat and efficient language.
It's not very exciting, but I'd consider using the built in script hosting support. I used to do this in Native C++ with no problem, and .Net seems to have the same thing wrapped (admit the article is old) at http://msdn.microsoft.com/en-us/library/ms974577.aspx
If you want other people to write scripts for your system, then PowerShell might be a good choice. From my observations, people are less resistant to learning PowerShell than to learning, say, Python. Probably that's because PowerShell comes with the Microsoft stamp of approval. In a Microsoft shop, Python may be seen as too exotic, and Ruby even more so.
It's easy to make the case that everyone in the Microsoft world---be they programmers, QA people, support people, power users---would benefit by learning PowerShell. It's easy to make the case that learning PowerShell would be aligned with their own self-interest. cmd.exe will be obsolete. Everyone will learn PowerShell sooner or later.
That's why I think PowerShell would be a good default choice.
Much of course depends on the particular requirements however.
A good test might be to write a typical script in each of the candidate languages, and see which one is the most natural. It sounds like PowerShell would be a good fit for what you describe. But if the scripts are very "algorithmic", then Python or Ruby would be better. If speed is a requirement, I believe IronPython is much faster than PowerShell. (I don't know about IronRuby.) Etc.
Update: I just had a profound thought :) An embedded scripting language is part of the user interface. That is, it's a UI issue. You should choose which one to use from the user's point of view.
Related
I want to make an application that executes a remote script. The user can create a script (probabily a LUA script) then stores it in the server. Then he can uses an API for execute the script. I was thinking that API could be a webservice.
So my questions are:
I need high performance to execute the script. So my first choice was LUA script. Someone has another sugestion?
Cause I need high perfomance, I was thinking if the webservice is the best solution. Maybe I could create a TCP/IP Windows Service that hold the users request. It is important to say that I will have many user executing scripts at the same time. So I will have a concurrency problem.
My scripts will query in a database. I will use Tokyo Cabinet or Tokio Tyrant. I think Tokio Tyrant is the only solution cause I will have many requests. For perfomance, Do I need to make a connection pooling? Is there anyway to share variables between webservices requests?
To make the webservice or the Windows service i was thinking to use C++.
Can someone help with these questions?
thanks
Lua is pretty high performance for a scripting language, especially if you use LuaJIT or something similar.
You speak of high performance. How much are we speaking about? Say you have a very simple webservice that executes scripts it receives via POST, then probably the HTTP overhead is comparably small when compared to the Lua compile, environment setup & execution time.
About the database I cannot tell you anything. There's many possibilities to do pooling and this also depends on how you execute the Lua scripts. Are they running in a common environment? One per session? One per request?
C++ surely is a good choice to host Lua, because Lua fits in pretty well. Though there are other good language bindings as well.
But keep in mind that your job is not over by just sandboxing scripts. User submitted scripts can do a lot other Bad Things(TM), intentionally or by mistake, like allocating a lot of memory or hogging the CPU. In Lua (and I think this is true of many, if not all, sandboxed environments) you cannot do much about this, except killing the offending instance or, if you disallowed using coroutines in your sandbox, yield out of the offending coroutine and do something smarter.
What do you prefer (from your developer's point of view) when it comes to implement a business process?
A Business Process Management System (BPMS) or just your favorite IDE with the needed tools and frameworks (a reporting tool for example)?
What is from your point of view the greatest Benefit of a BPMS compared to an IDE with your personal tools and frameworks?
OK. Maybe I should be more specific... I got to know one specific BPMS which should make it easy to implement a business process by configuring rules. But for me as a developer it is hard to work with the system. I would like to work with text files which I can refactor and I would like to be able to choose the right technology or framework for the job I have to do. Instead the system forces me to configure.
There are rules where I can use java, but even then I have to stick to the systems editor without intellisense etc.
So this leads me to the answer of my own question - I would like to use the tools I am used to instead of having to learn how to work with a BPMS (at least the one I know) because it limits me more than it helps. The BPMS I know is a framework from which it is hard to escape! At this time, I would prefer a framework like Grail over any BPMS I know.
So maybe the more specific question is: do you feel the same or are there BPMSes which support you in beeing a developer and think like a developer or do most of them force you to do your job a different way?
In my experience the development environments provided by BPMS systems are third rate, unproductive, and practically force you to write hard to maintain, poorly designed code (due to their limitations). Almost all the "features" (UI, integrations, etc) provided by the BPMS system I'm familiar with (the one sold by that company named for its database) were not worth the money we paid.
If you're forced to use BPMS, as a developer, my advice would be to build as much of your application in a conventional development environment, such as Java or .Net, build as little as possible in the BPMS environment itself, and integrate the two. The only things that should go in the BPMS is the minimum to make the business process work.
Not sure what exactly you ask, but the choice BPM vs. plain programming will depend on the requirements. A "business process" is a relatively vague term in software engineering.
Here are a few criterion to evaluate your needs:
complexity of the rules - Are the decisions/rules embodied in your process simple, complicated, configurable, hard-coded?
volatility of the process - How frequently does your process change? Who should be able to make the change?
integration need - Is your process realized using multiple heterogenous services, or is all implemented in the same language?
synchronous/asynchrounous - Is your process "long-running" with the need to handle asynchronous actions?
human tasks - Does your process involves human interaction, with task being assigned/routed to people according to their roles/responsibilities?
monitoring of the process - What is the level of control you want on the existing process instances being executed? Do you need to audit the actions, etc. ?
error handling - Depending on the previous points, how do you plan to deal with errors, or retry of faulty process execution?
Depending on the answer to these questions, you may realize that your process is closer to a simple state chart with a few actions and decisions that can be executed in a sequence, or you may realize that you need something more elaborated, and that you don't want to re-implement all that yourself.
Between plain programming and a full-fledge BPM solution (e.g. Oracle BPM suite which contains BPEL, rule engine, etc.), there are intermediate solutions such as jBPM or Windows Workflow Foundation and probably a lot of others. These intermediate solution are frequently good trade-off.
I have worked with Biztalk in the past and more recently with JBPM. My opinion is biased against BPMs for the following reasons:
Steep learning curve : To make a process work, I have to understand how the system and the editor works. It is hard enough for a developer to understand the system, let alone a business user. The drag and drop and visual representation is a great demo tool. It certainly impresses managers (who ultimately pay for it), but a developer's productivity just drops.
Non developers changing the workflow : I haven't seen one BPM solution do it flawlessly. Though it doesn't look like code, right click on the box and you do have to put some code, otherwise it is not going to work. So you definitely need a developer to do it. The best part is that it is neither developer friendly nor business user friendly, just demo user friendly.
Testablity and refactoring : It is virtually impossible to test drive a BPMS. You do have 'unit test frameworks' advertised, but most of them are hacks and hard to use. Recently I tried the JBPM one; I ended up writing a lot of glue code and fake workflow handlers to make it work. The deal breaker for me though is refactoring. If the business radically changes it's mind about how a business process should look, then good luck re-arranging the boxes, because just re-arranging them won't work, all the variables bound to the boxes also need to be re-arranged. I would prefer the power of the IDE and tests to refactor my business process.
If your application has workflow, then you could try a workflow library (with or without persistent state). It will still manage your workflows without all the bloat that comes with a BPM. If a business user needs to understand the code, then let the business prepare good process flowcharts and translate them into good domain driven code. Use cucumber style acceptance tests to make bring the developers and business together. A BPM is just something that tries to do too many things and ends up doing all those things badly.
BPMS-- a lot of common business case, use case are already implemented. So you just have to know how to use it. For common workflow, you don't even need to write a single line of code, though mostly you would have to write some scripts to cover things that are not yet implemented.
Plain programming-- just use the IDE to hack out the code. The positive side: more control. The negative? A lot of times are spent on rewriting boilerplate code. And you have to maintain them.
So in a nutshell, I would prefer a Business Process Management System. One that I would recommend is ProcessMaker. It features an intuitive process designer that allows you to design workflow with drag and drop. And you can always write trigger to extend the process functionalities. It's open source as well.
I'm playing WoW for about 2 years and I was quite curious about Lua which is used to write addons. Since what I've read so far about Lua was "fast", "light" and "this is great", I was wondering how and when to use it.
What is the typical situation where you will need to embed a script language like Lua in a system ?
When you need end users to be able to define/change the system without requiring the system to rewritten. It's used in games to allow extensions or to allow the main game engine to remain unchanged, while allow content to be changed.
Embedded scripting languages work well for storing configuration information as well. Last I checked, the Mozilla family all use JavaScript for their config information.
Next up, they are great for developing plugins. You can create a custom API to expose to the plugin developers, and the plugin developers gain a lot of freedom from having an entire language to work with.
Another is when flat files aren't expressive enough. If you want to write data driven apps where behavior is parameterized, you'll get really tired of long strings of conditionals testing for config combinations. When this happens, you're better off writing the rules AND their evaluation into your config.
This topic gets some coverage in the book Pragramtic Programmer.
Lua is:
Lightweight
Easy to integrate, even in an asynchronized environment such as a game
Easy to learn for non-programmer staff such as integrators, designers and artists
Since games usually require all those qualities, Lua is mostly used there. Other sitation could be any application that needs some scripting functionality, but developers often opt for a little more heavy weight solution such as .Net or python.
In addition to the scripting and configurability cases mentioned, I would simply state that Lua+C (or Lua+C++) is a perfect match for any software development. It allows one to make an engine/usage interface where engine is done in C/C++ and the behaviour or customization done in Lua.
OS X Cocoa has Objective-C (C and Smalltalk amalgam, where language changes by the line). I find Lua+C similar, only the language changes by a source file, which to me is a better abstraction.
The reasons why you would not want to use Lua are also noteworthy. Because it hardly has a good debugger. Then again, people hardly seem to need one either. :)
a scripting language like Lua can also be used if you have to change code (with immediate effect) while the application is running. one may not see this in wow, because as far as i remember the code is loaded at the start (and not rechecked and reloaded while running).
but think of another example: webserver and scripting language - (thankfully) you can change your php code without having to recompile apache or restart apache.
steve yegge did that thing for his own mmorpg engine powering wyvern, using jython or rhino and javascript (can't remember). he wrote the core engine in java, but the program logic in python/javascript.
the effect of this is:
he doesn't have to restart the core engine when changing the scripts, because that would disconnect all the players
he can let others do the simpler programming like defining new items and monsters without exposing all the critical code to them
sandboxing: if an error happens inside the script, you may be able to handle it gracefully without endangering the surrounding application
Rapid development for application with real-time constraints. Computer games are one of these ;-)
It's a valid solution if you want to allow third parties to develop plug-ins or mods for your software.
You could implement an API in whatever language you are using, but a script language like LUA tends to be more simple and accessible for casual developers.
In addition to all the excellent reasons mentioned by others, Embedding Lua in C is very helpful when you need to manipulate text, work with files, or just need a higher level language. Lua has lots of nifty feature (Tables, functions are first class values, lots of other good stuff). Also, while lua isn't as fast as C or C++, it's pretty quick for an interpreted language.
For starters I should let you guys know what I'm trying to do. The project I'm working on has a requirement that requires a custom scripting system to be built. This will be used by non-programmers who are using the application and should be as close to natural language as possible. An example would be if the user needs to run a custom simulation and plot the output, the code they would write would need to look like
variable input1 is 10;
variable input2 is 20;
variable value1 is AVERAGE(input1, input2);
variable condition1 is true;
if condition1 then PLOT(value1);
Might not make a lot of sense, but its just an example. AVERAGE and PLOT are functions we'd like to define, they shouldn't be allowed to change them or really even see how they work. Is something like this possible with DLR? If not what other options would we have(start with ANTRL to define the grammar and then move on?)? In the future this may need to run using XBAP and WPF too, so this is also something we need to consider, but haven't seen much if anything on dlr & xbap. Thanks, and hopefully this all makes sense.
Lua is not an option as it is to different from what they are already accustomed to.
Ralf, its going to reactive, and to be honest the timeframe for when the results should get back to the user may be 1/100 of a second all the way up to 2 weeks or a month(very complex mathematical functions).
Basically they already have a system they purchased that does some of what they need, and included a custom scripting language that does what I mentioned above and they don't want to have to learn a new one, they basically just want us to copy it and add functionality. I think I'll just start with ANTRL and go from there.
Lua
it's small, fast, easy to embed, portable, extensible, and fun!
Lua is definitly the best choice for soft real-time system (like computer games).
See http://shootout.alioth.debian.org/ for detailed benchmarks.
However, last time I checked, Lua used a mark-and-sweep garbage collector which can lead to deadline-violation and non-deterministic jitter in real-time systems.
I believe that you could use theoretically use the DLR, but I'm unsure about support in an XBAP (partially trusted?) scenario.
If you host the DLR you would quickly be able to take advantage of IronRuby or IronPython scripting. You would want to look at these implementations when creating your own language implementation. If you post your question to the IronPython mailing list I'm sure you would get a better reply around the XBAP scenario, and some of the developers there created ToyScript.
What kind of real-time requirement are you trying to fulfill? Is the simulation a hard real-time simulation (some kind of hardware-in-the-loop simulation ==> deadline is less than 1/1000 second)?
Or do you want the scripting-system to be "reactive" to user-input ==> 1/10 should be sufficient.
I am no expert regarding MS DLR, but as far as I know, it does not support hard real-time systems. You may want to take a look at the real-time specification for Java (RTSJ)
Firstly I think that defining your own language is not the way to go.
Primarily because the biggest productivity gains you can get for programmers or non-programmers are the development tools. You (and 99.9% of the rest of us) are not going to write tools as good as what is out their.
Language design is hard.
Language support and documentation, also hard
I would recommend looking for a pre-built solution. If you could find a language that can lock down some functionality, that would be a good starting point. MatLab would be the first that comes to my mind.
Lastly, ditch the natural language part, BASIC, COBOL and YA-TDWTF-Lang all tried and failed at it.
Full disclosure: I work for a company that is developing a generalized domain specific language "system". It's targeted at data-in/text-out applications so it's not apropos and it's not yet to beta. The result is I'm somewhat knowledgeable and biased.
I'm a Java programmer, and I like my compiler, static analysis tools and unit testing frameworks as tools that help me quickly deliver robust and efficient code. The JRE is pretty much everywhere I would work, too.
Given that situation, I can't see a reason why I would ever choose to use shell scripting, vb scripting etc, no matter how small the task is if I wear one of my other hats like my cool black sysadmin fedora.
I don't wear the other hats too often, under what circumstances should I choose scripting over writing compiled code?
Whatever you think will be most efficient for you!
I had a co-worker who seemed to use a different language for every task; Perl for quick text processing, PHP for small internal web applications, .NET for our main product, cygwin for filesystem stuff. He preferred to use the technology which was most specific to the task at hand.
Personally, I find that context switching between technologies is painful. My day-to-day work is in .NET, so that's pretty much the terms I think in. For most tasks I find it more efficient to knock something up in C# using SnippetCompiler than I would to hack around in PowerShell or a scripting environment.
If you are comfortable with Java, and the JRE is everywhere you work, then I would say keep using it. There are, however, languages like perl and python that are particularly suited to quickly solving problems. I would suggest learning either perl or python, and then use your judgement on when to use it.
If I have a small problem that I'd like to solve quickly, I tend to use a scripting language. The code tax is smaller, and, for me at least, the result comes faster.
I would say where it makes sense. If it's going to take you longer to open up your IDE, compile the script, etc. than it would to edit a script file and be done with it than use script file. If you're not going to be changing the thing often and are quicker at Java coding then go that route :)
It is usually quicker to write scripts than compiled programmes. You don't have to worry so much about portability between different platforms and environments. A shell script will run pretty much every where on most platforms. Because you're a java developer and you mention that you have java everywhere you might look at groovy (http://groovy.codehaus.org/). It is a scripting language written in java with the ability to use java libraries.
The way I see it (others disagree) all your code needs to be maintainable. The smallest useful collection of code is that which a single person maintains. Even that benefits from the language and tools you mentioned.
However, there may obviously be tasks where specialised languages are more advantageous than a single general purpose language.
If you can write it quicker in Java, then go for it.
Just try and be aware of what the various scripting languages can do.
e.g. Don't make a full blown Java app when you can do the same with a bash one-liner.
Weigh the importance of the tool against popping open a text editor for a quick edit vs. opening IDE, recompiling, redeploying, etc.
Of course, the prime directive should be to "use whatever you're comfortable with." If Java is getting the job done right and on time, stick to it. But a lot of the scripting languages could save you some time because they're attuned to different problems. If you're using regular expressions, the scripting languages are a good fit. If you're dropping into shell commands, scripts are nice.
I tend to use Ruby scripts whenever I'm writing something that's small, because it's quick to write, easy to maintain, and (with Gems) easy to bolt on additional functionality without needed to use JARs or anything. Your milage will, of course, vary.
At the end of the day this is a question that only you can answer for yourself. Based on the fact that you said "I can't see a reason why I would ever choose to use shell scripting , ..." then it's probably the case that you should never choose it right now.
But if I were you I would pick a scripting language like python, ruby or perl and start trying to solve some of these small problems with this language. Over time you will start to get a feel for when it is more appropriate to write a quick script than build a full-blown solution.
I use scripting languages for writing programs which are not expected to be maintained beyond few executions. Most of these languages are light on boiler-plate syntax and do have a REPL. Both these features enable rapid prototyping.
Since you already know Java, you can try JVM languages like Groovy, JRuby, BeanShell etc. Scala has much lighter syntax than Java, has a REPL, is statically typed and runs on the JVM - you might give that a shot as well.