I have a MATLAB application that I want to execute on a linux box with inputs from a web server. Requests to the server would all be from the local network.
Searching for different solutions, I've seen recommendations to host a Django server that serves an HTML form where users could input all the various data needed by the application. When a user fills out the form and submits it, the data would be sent through an API to the MATLAB application, which would serve up the report in a network shared drive.
Would this work well? Is there a different/easier solution available?
Need more details to know if this would "work well". But in terms of the general outline you presented, seems feasible.
When you say "the data would be sent through an API to the MATLAB application", what exactly do you mean here? What API are we talking about? And what is "the Matlab application"? Do you mean just installing regular Matlab on this server machine, and then having the Django or other web application server run the matlab command to run a Matlab program, running as a distinct process (corresponding to a single matlab -batch execution, probably?) that services that? Two issues here: One, Matlab is a large program with a slow startup time. Matlab Production Server and similar solutions handle this by maintaining a pool of already-running "warmed up" Matlab worker processes to service incoming requests. Two, licensing: the "regular" Matlab licenses are aimed towards interactive use by humans; running Matlab like that on the server side to handle requests for a web app used by multiple humans may not be covered. Talk to your organization's lawyer or IT licensing expert before doing this.
#Will is right here: The Matlab Production Server is the product or "solution" that MathWorks provides for this scenario. And it's relatively easy to use. But ain't cheap. (On the other hand, when you're talking about Matlab, what is?)
If you have someone who can do a bit of system programming for you, there's a more affordable alternative: use the Matlab Compiler to build your Matlab code into a "CTF" DLL, and write a thin custom server wrapper on top of that, which can accept service calls for the particular Matlab things you need done, and dispatches them to your code. (Running that in a pool of multiple processes, if you want to be able to service multiple concurrent clients.) "Compiled" Matlab libraries that run against the Matlab Runtime do not require any additional licenses for their runtime execution.
Big questions here are: Do you want this to Go Fast? How many clients are you going to have, and how often are they going to be sending requests? What kind of data will be contained in the inputs and outputs to this Matlab code?
Have a look at the -batch option to the matlab command. Have a look at the various deployment options supported by the Matlab Compiler. And talk to your organization's lawyer.
If you decide to go the matlab -batch route, you probably do not want to pass the inputs to your Matlab code as command-line arguments. Command lines and environment variables only pass simple strings, and parsing those sucks, especially once you get in to nontrivial numerics. Bundle up all your inputs as JSON files, MAT files, or something similar, and then pass just a reference to those files (or SQL blobs, or similar) on the command line.
Also, depending on what your Matlab code is like, GNU Octave (https://gnu.org/software/octave/index) may be an option for you. Octave is many years behind Matlab in terms of functionality and stability, and doesn't have equivalents of all the Matlab Toolboxes, so it isn't a drop-in replacement in general terms. But for simple stuff, it works. And it is unencumbered by licensing, and has faster startup times in command-line mode.
An easier solution (though possibly not the cheapest if your existing MATLAB license doesn't already cover it) is to use MATLAB Production Server which basically exists for this category of problem. It has a RESTful API that would straightforwardly handle your user input use case.
Related
I am looking for something along the same lines as this article: "Go With Go(Lang): Features and Testing Frameworks", featuring Clojure/Clojurescript. Similarly, any book(s) that contain examples of Clojure/Clojurescript being used as to create devops/test-automation applications, would also help.
I am the developer behind spire, a clojure DSL for idempotent machine provisioning over ssh. I built it using GraalVM and sci. I have also used it for automated testing.
Presently I am working on making it a babashka pod, so soon you will be able to use its functionality directly from babashka.
I do virtually all my AWS manipulations using aws-api. The dev-ops experience via clojure is much better than using bash/shell scripts and the AWS CLI.
Your question is a bit vague, but Clojure is IMHO the best general-purpose computer language (no surprise since you asked on a Clojure site!).
One solution that might work for you is the Babashka tool. It allows you to turn Clojure code in to a stand-alone executable. So, instead of using quirky, fragile bash scripts, you can write a robust & powerful Clojure program that has the instant startup & low memory usage of a Go or C program.
For more general purposes, you can write a complete general-purpose Clojure program and interop with the outside O.S. with a function like tupelo.misc/shell-cmd or the original Clojure version from clojure.java.shell. I have used this to drive AWS CLI functions, for example listing, cloning, and deleting RDS database instances.
The sky is the limit (along with your imagination).
Update
I forgot to mention GraalVM. It allows you to compile Clojure code into a static executable (the same as produced by a C/C++ program). Here is a Clojure/GraalVM demo project that creates a "Hello World" program even faster and more memory efficient than a Python script! Enjoy!
I am writing an open source cross-platform application written in C++ that targets Windows, Mac, and Linux on x86 CPUs. The application produces a stream of data (integers) that needs to be validated, and my application will perform actions depending on the validation result. There are multiple validators, which we shall call "modules", and they can be swapped out for one another.
Anybody can write and share modules with other users, so my application has to ensure that maliciously-written modules cannot harm the user in any way (perhaps except via high CPU usage, in which case my application should be able to kill the module after some amount of time - this can be done by using a surrogate process). Furthermore, the stream of data is being sent at a high rate (up to 100kB/s).
Fortunately, the code in these modules are usually simple arithmetic operations on data in the stream (usually processing each incoming integer in constant time), and they do not need to make any system calls (not even heap allocation).
I've considered the following possibilities (all of them with some drawbacks):
Kernel-based sandboxing
On Linux, we can use secure computing (seccomp), which prevents a process from making any system calls except for reading and writing with already-open file destriptors. Module creators would write their modules as a single function that takes in input and output file descriptors (in a language like C or C++) and compile it into a shared object, then distribute that shared object.
My application will probably prepare input and output file descriptors, then fork() itself or exec() a surrogate process, and this child process uses dlopen() and dlsym() to get a pointer to the untrusted function. Then strict secure computing mode will be enabled, before executing the untrusted function.
Drawbacks: There's the problem that dlopen() will actually run the constructor function from the shared library. This would have to be properly sandboxed as well, and I can't think of a way to do so. Also, of course, this thing will only work on Linux. As far as I know, there is no way to ban WinNT system calls on Windows, so a similar solution on Windows won't be very secure.
Application-level sandboxing
[[ Any form of application-level sandboxing means that we cannot run untrusted machine code of any form. An untrusted function can overwrite its return value or data outside its call stack, thereby compromising the whole application (and effectively acquiring any permissions that the original application had). ]]
Make modules use a simple scripting language that does not support any system calls - just pure arithmetic operations and perhaps the ability to read an input stream. My application would contain an interpreter for this language.
Drawbacks: Unfortunately I have not found this scripting language. Many scripting languages have extensive functionalities (e.g. Python) and a sandbox (e.g. PyPy's sandbox) simply filters OS system calls. I would be shipping a lot of useless interpreter code with my application, and it arguably is more prone to security issues due to bugs in the intepreter than a language with simply no functionality to do things other than simple calculations and control flow instructions (basically a function that does not make any system calls). Furthermore, marshalling the data between C++ (machine code) and the scripting language is usually a slow process.
Distribute modules with a 'safe' compiled language that again does not support any system calls. My application would contain a JIT for this language.
Marshalling won't be necessary because my application would call into the JITted machine code of the untrusted module, so performance across this boundary should be fast. The untrusted module now won't be able to corrupt the stack, attempt return-oriented programming, or perform any other malicious actions, due to the language restrictions and checks of the 'safe' language. WebAssembly is the first and only language that comes to mind (if it can be called a language). (As far as I can tell, WebAssembly seems to provide the security guarantees for my use case, right?)
Drawbacks: The existing implementations of WebAssembly seem to be all browser-based, so I would have to steal an implementation from an open source browser. This does seem like a lot of work, considering that I would have to uncouple it from all the JavaScript and other browser bits. However, a standalone WebAssembly JIT based on LLVM seems to be under development.
Question:
What is the best way to execute an untrusted function efficiently that works on Windows, Mac, and Linux?
Right now, I think that the scripting language way would probably be the safest, and be the easiest for module writers. But for a more efficient solution, WebAssembly is probably better. Am I right, or are there better or easier solutions that I have not thought of?
(Remark: I think several pairs of tags used in this question have never been seen together before!)
Regarding WebAssembly:
Unfortunately, there is no production-quality stand-alone implementation yet. I expect some to show up in the future, but it hasn't happened yet.
For historical reasons, existing production implementations are all part of a JavaScript VM. Fortunately, none of these VMs is tied to a browser. If you don't mind including some unused JS baggage, you can embed them as they are (ripping out the JS would be very hard). One problem, though, is that these VMs don't yet provide embedding interfaces for Wasm specifically. You have to go through JS, which is stupid.
There is an initial design for a C and C++ API for WebAssembly, which would give direct access to an embedded Wasm VM. It is meant to be VM-neutral, i.e., could be implemented by any existing VM (the repo contains a prototype implementation on top of V8). This may evolve into a standard, but I cannot promise any timeline. Right now it's only for the brave.
I am simply reading information off the interwebs, currently the cmake about page, and I need the information to fill in the gaps, it helps to see the big picture.
Surely the answer is straightforward, I hope. What is a native build environment?
Context: I need to know how to build software on my machine (CodeBlocks, etc), why I need to do this, the advantages of doing this, etc. But first, I need to know every piece of jargon I come across, and I could not find any explanations about exactly what a "Native build environment" is, although I can speculate to some degree.
"Native" as in "runs directly in the host operating system" and not "runs in a virtual machine or emulator."
The particular point that CMake's about page is trying to convey is the manner in which CMake achieves cross-platform functionality: specifically not by virtualizing, but by cooperating/collaborating directly with the host system, and the "normal" ways the host system is used to doing things.
Is the build environment then just the directory holding all the garbage needed for a compiler to build the software then?
That's an oversimplification – there's nothing to say that it's a single directory – but more or less, yes. The term is not jargon, it literally means "the state of the world" (aka, environment) needed for the build.
What would you call the other thing then, Non-native?
Sure, or virtualized, or emulated, or whatever other intermediate layer has been added.
Why do we need the distinction as well?
Why not? It's useful to have a concise, clear, simple term so we can communicate precisely and with minimal confusion and ambiguity.
Why 'non-native'? If you haven't already figured this out - there is something called cross-compilation.
Simply put, if I don't have access to target hardware (or an equivalent virtual machine) on which the software needs to run, how do I develop this software on my host and package it to run on that target?
Cross-compilation addresses this by providing necessary tools that perform a translation (or other important low-level stuff) to give you the final software. Such an environment to develop software is called non-native.
Well, I believe we need the term to state the technique.
I would like to call MQL4 or MQL5 function from my own imported DLL in Metatrader.
Is it possible?
Forest,
As far as i have experienced during the past 2 years working with MetaTrader, there is no real way to call MQL functions from an external DLL. But there are some custom built APIs that closely resemble to what you want to achieve:
MT4 API
MetaTrader™ Java / .Net API
These APIs do somewhat allow you to use MQL functionality out-of-the-box
Principle
After several hundred man*years in the FX domain, there is another approach to orchestrate a smooth and elegant MT4 Terminal co-operation with other processes than to try to push water up the hill or than to pay USD500+ for a kit, that will stop working right upon the next shock once Build 524-> Build 562->Build 586->Build 600->Build 609->Build 624->... moves again
A non-existent toy
Yes, MT4 architecture does not expose it's own interface to allow self to be "disturbed" by an undeterministic obligation to handle external low-level calls via DLL et al.
How to fix it
Nevertheless, it is possible to reverse the architecture and make MT4 Terminal act as a lightweight thin-Client, operating a smart messaging library, trough which the MT4 functions are being exposed for a remote call ( RPC ).
Example
This way a Python Node may collect MT4 data for numerical processing,
same way a PHP Node may in parallel handle remote-syslog-s,
same way a C++ Node may integrate another task,
same way another Python Node may act as a CLI terminal interface with a Custom-specific scripting-syntax language to command MetaTrader-side activities via command-line / stdio
simply -- whatever your application infrastructure needs can be done this way
( One may even improve a poor real-time features of the native MT4 threads to gain a much better soft-real-time predictability and a low-latency massively parallel architecture .. and still be on a safer-side, protected from being torpedoed by any next "new"-MQL4 )
nota bene: just to imagine the invisible threat, the headbang collision in "new"-MQL4.56789 is, besides others, that string, while being syntax-proposed as string, is not in fact a string but a struct and all your previous DLL-related work simply has to be re-worked and wrapped-around to emulate a string-as-struct or a new DLL-interface has to be designed for cases, which return a value in a buffered ArrayOfBYTEs, which MQL4.56789 side can receive and process, but which it cannot free on it's own and memory leaks.
If it's acceptable for your DLL to be a .NET DLL, then you could try
this MT4 .NET integration library called NQuotes.
With this library it's possible to access any MQL4 function from your DLL.
A project is looming whereby some code that I will be writing may be deployed on any hardware that potential clients happen to have. Its a business application that will be running 24/7 so I envisage that most of the host machines will be server type boxes but smaller clients might, for example, just have a simple PC.
A few more details about the code I will be writing:
There will be no GUI.
It will need to communicate with another bespoke 'black box' device over an Ethernet network.
It will need to communicate with a MySQL database somewhere on the network.
I don't have any performance concerns as a) the number of communications with the black box will be small, around 1 per second, and the amount of data exchanged will be tiny (around 1K each time), b) the number of read/writes with the database will be small, around 5 per minute, and again the amount of data exchanged will be tiny and c) the processing that needs to be performed is fairly simplistic.
Nothing I'm doing is very 'close to the metal' so I don't want to use languages that are too low level. Ease of development and ease of deployment are my main priorities.
I'm not expecting there to be a perfect solution so I can live with things like, for example, having to have slightly different configuration files for Windows machines than for Linux boxes etc. I would like to avoid having to compile the software for each host machine if possible though.
I would value your thoughts as to which development language you think is most suitable.
Cheers,
Jim
I'd go with a decent scripting language such as Python, Perl or Ruby personally. All of those have decent library support, can communicate easily with both local and remote MySQL databases and are pretty platform independent.
The first thing we need to know is what language skills you already have? This is likely to be a fairly big determiner of what choice you would ideally make.
If I was doing this I'd suggest Java for a couple of reasons:
It will run almost anywhere and meet the requirements you've outlined.
Its not an esoteric language so there will be plenty of developers.
I already know how to program in it!
Probably the most extensive library ecosystem of any of the development platforms.
Also note that you could write it in another language on the JVM if your more comfortable with Ruby or Python.
Sounds like Perl or Python would fit the bill perfectly. Which one you choose would depend on the expertise of the people building and supporting the system.
On the subject of scripting languages versus Java, I have been disappointed with developing command line tools using Java. You can't directly execute them, you have to (1) compile them and (2) write a shell script to execute the jar file, this script may differ between platforms. I recommend Python because it runs anywhere and it's got a great SQL library, mysql-python. The library is ready to use on Windows and Linux. Python also has a lot less boilerplate, you'll write fewer lines of code to do the same thing.
EDIT: when I talked about JARs being executable or not, I was talking about whether they are directly executable be the OS. You can, of course, double click on them to run them if your file manager is set up to do so. But when you're in a terminal window and you want to run a java program, you have to "java -jar myapp.jar" instead of the usual "./myapp.jar". In Python one just runs "./myapp.py" and doesn't have to worry about compiling or class paths.
If all platforms are standard PCs (or at least run Linux), then Python should be considered. You can compile it yourself if no package exists for your version. Also, you can strip the standard library easily from things that aren't available and which you don't need (sound support, for example).
Python doesn't need lots of resources, it's easy to learn and read.
If you know Perl, you can try that. If you don't use Perl on a daily basis, then don't. The Perl syntax is hard to remember and after a week, you'll wonder what the code did, even if you wrote it yourself.
Perl may be of help to you as it is available for many platforms and you can get almost any functionality by simply installing modules from CPAN.
Python or Java. They both are easy to deploy on both the server environments and the desktop environments you mention - i.e., Linux/Solaris and Windows.
Perl is also a nice choice, but it depends on how well you know Perl, how well other people that will maintain your code know Perl, and number of desktop users that are savvy enough to handle an install of the Windows Perl version(s).
As Java supports Python via Jython, I'd go with a JVM requirement myself, but I'd personally go with a Java application all the way for such a system you describe.
I would say use C or C++. They are platform independant, though you will have to compile for each platform.
Or use Java. That runs in a Virtual Machine so is truely cross platform and not a slow level as C.