I am a career .NET developer, but have recently been delving into the LAMP universe.
I have an Joomla/VirtueMart ecommerce site near ready to launch. The vendor's xml datafeed must undergo transformation before it can be imported into the products database.
I wrote a .NET console app that can easily perform the transform and upload to the site. I am convinced there is a better way.
So I have been looking at cgi scripts. Just from what I have read it seems the only way to execute a cgi script is through an http request. Is there a way to schedule a cgi-script to run at a specific time?
Also, which language works best for transforming xml? C, Perl, or Python?
CGI is basically just an interface defined to execute arbitrary programs on a server by way of a HTTP request, nothing more and nothing less.
How you can execute a program on a server at a specific time depends on the server; both Windows and Linux-based systems usually have a scheduler service. In Linux land it's called cron, and a scheduled job is called a cronjob. If there is no other mechanism provided by whoever hosts your application, you can define jobs using the crontab utility on the server (you'll need shell access for that); its documentation includes (somewhat inaccessible, but there you go) information about what a crontab should look like. There are tons of tutorials on the web, too.
an XSLT transformation is probably your best bet for turning XML files into things you can easily import. An XSLT transform can be processed in any language of your choice.
Related
So I have this bit of a project planned for Windows Store and Android. Basically, a networking multi-tool coupled with a scripting engine to implement protocols and behavior. Ideal uses being things like "my embedded device uses this simplistic network protocol. I'd like to quickly prototype a way to control it from my tablet".
It's my understanding that the Android market should have no problem with this. However, the Windows Store policy includes a vague clause concerning remote code execution
3.9 All app logic must originate from, and reside in, your app package Your app must not attempt to change or extend the packaged content
through any form of dynamic inclusion of code or data that changes how
the application interacts with the Windows Runtime, or behaves with
regard to Store policy. It is not permissible, for example, to
download a remote script and subsequently execute that script in the
local context of your app package.
Of course, the scripting engine will be sandboxed and such and should be "safe"(completely intepreted, no reflection), but does it violate this policy?
If you build in your scripting engine, and only run local scripts, you will be good. However, if you were thinking to have a repository of scripts that could be downloaded and subsequently run, that would be in violation of the policy as we understand it.
Unfortunately I don't think anyone but someone on that team can answer that (or someone with direct experience in that) because of the closeness to the legal language. Have you tried the Windows Store Appl Publishing forum at: http://social.msdn.microsoft.com/Forums/en-US/windowsstore/threads
In the context of scripting engine example given, unless the app modifies the scripting engine after deployment on user's system such that the representation of protocol/behavior (the script artifact's format) is made to change then it'll be policy violation. Its as if you submit Python interpreter, and at some point in time it abruptly moves onto interpreting ecmascript.
If you look at the Linux ecosystem (especially the Ubuntu and Alestic EC2 images) there is a common technique where the VMs are pre-configured to look at the EC2 user-data and use it as a boot script. The nice thing about this approach is that you can write a boot script that further provisions your machine, allowing you to avoid making a new image every time your software that runs on the machine changes.
I want to do the same thing for Windows, but given that I'm an Mac and Linux guy, I'm a bit lost on where to start. My requirements are:
This must run on Windows Server 2008
A bootstrap script needs to start when the machine boots up, read the user-data file by pulling down the contents http://169.254.169.254/1.0/user-data
The bootstap script then needs to run the contents of that file as if it were a script
The script embedded in the user-data needs to run in such a way that it has access to the desktop environment (ie: it can launch a browser, etc).
I'm not quite sure how services work in Windows or if I need to enable auto-login, so any advice here would be appreciated. The ultimate goal is to run a Java program that launches some custom software that in turn launches a web browser (IE, Firefox, etc) and is capable of taking screenshots.
The screenshot part is interesting, because in the past when I've tried this the only way I could get something other than a black screen was to have UltraVNC or RealVNC boot up as a service, though I don't know why that helped.
I'm looking for answers to three specific questions, as well as any general advice:
Should I be focussing on a Windows service or auto-login + bat file in the "Startup" folder?
If I use a Windows service, is there anything special that I need to do to make sure desktop access and/or screenshots are available?
Do you recommend any tools for common Linux commands, like curl or wget? Last time I used Windows I used Cygwin a lot, but is there something more appropriate to use here?
I have not tried auto-login on Windows instances in EC2, but here's the support document on how to enable it.
We boot-strap our Windows instances using a custom AMI with a custom Windows 'install' service already installed. The boot-strap installer reads a URL from user-data at startup. The URL points to a ZIP file stored in S3. The installer then downloads, un-zips, and executes the actual application installer -- in our case a simple CMD fie.
This setups allows us to have one base AMI and then be able to easily overlay 15+ different application configurations (without having to rebuild the AMI). If you only have one application configuration this may be overkill for your situation.
The only trouble we ran into was having our installer service start to early -- changing the service startup mode to "Automatic Delayed" fixed that issue.
We wrote our boot-strap installer in Java, launched via YAJSW, because we're comfortable with it. If you just want a few simple Unix tools, most are available pre-compiled for Windows, for example wget.
For something completely different, you could try PsExec to configure the instance after it has booted.
You can try using RightScale's free developer account to create plain Powershell scripts and associate them with your Windows instances to run at boot time. The RightScale dashboard solves exactly the problems you are trying to solve above.
DISCLAIMER: I work for RightScale.
As for screen capture CutyCapt is a simple tool you can point at a URL and generate an image from.
Unxutils is a great solution for those looking for unix tools on Windows. It's got the wget.exe that you're looking for, however, using Powershell to download stuff is not so bad either:
$wc = new-object system.net.webclient
$wc.DownloadFile("http://stackoverflow.com","test.html")
If you can write a batch file to do your setup, then you can run it at startup of the vm by doing this:
1. Run REGEDT32.EXE.
2. Modify the following value within HKEY_CURRENT_USER:
Software\Microsoft\Windows NT\CurrentVersion\Winlogon\ParseAutoexec
1 = autoexec.bat is parsed
0 = autoexec.bat is not parsed
As an answer to #3, I would say that you can do just about anything in a batch file that you need which includes downloading from a ftp server (but not from a http server). I am really interested in this stuff and so if you have questions, try asking me.
If you use Elastic Beanstalks you can use this:
Customizing the Software on EC2 Instances Running Windows
It uses YAML formatting standards, e.g.
packages:
msi:
mysql: http://dev.mysql.com/get/Downloads/Connector-Net/mysql-connector-net-6.6.5.msi/from/http://cdn.mysql.com/
or
sources:
"c:/myproject/myapp": http://s3.amazonaws.com/mybucket/myobject.zip
I know this is a little bit late to help out with the original post but for anyone who is still reading this one solution is to use the http://cloudinitnet.codeplex.com/ project. The service is easily installed using a powershell script and will create a local administrator account to use while running.
The goal for this project was to replace the Cloud-Init project used in Amazon Linux and Ubuntu.
I'm looking for the equivalent of a URL shortening service such as http://bit.ly/ for an internal deployment in our organisation. Anyone know of any open source projects (especially Java ones) or commercial products which I can install internally rather than using an external service?
Thanks!
Shorty : http://get-shorty.com/
But there's several other url shortener .... most of them are in PHP/Mysql.
Don't know if a Java one exist.
http://monkeytooth.net/2010/12/htaccess-php-how-to-wordpress-slugs/
tells you the core basics of how to achieve the concept with PHP and Htaccess building up from there I can say would solely be on your own. However not all to hard a concept in general to build off of if you know php/mysql. That said your not likely to find anything directly built in JavaScript however using this with JavaScript again wouldn't be all that hard a concept. I say your not likely to find one JS based as you need some type of server-side script to communicate with a DB somewhere, where you have all your short URL identifiers, and JavaScript to my knowledge doesn't support directly at least database connectivity. You can go through any means of AJAX to communicate with a server-side script to then do what you want with the JavaScript though.
My current role requires me to setup environment which mimics the customer's and perform various checks to replicate and then analyze the problem.
Chances are, I often find working with Windows environments such like XP, Server 2003, Server 2008 is a bit painful without having the handly linux-based shell and some command-line progamming languages such as Perl.
Of course I can just install everything onto the new system and then start working, but it is a bit time-consuming and boring.
So I am wondering which is a better way of working around this?
I can for sure use Qemu to create a portable linux image which doesn't require any host system interference, even without the need of rebooting so to use it. The weakness of this is I have to figure out a way to transfer the files between hosting Windows and embedded Linux. The good part is that I can use all the weapons in Linux's arsenal.
Or I can start looking for a proper portable progamming language such like Movable Python, some variant of Perl or even Lua as a embedded language. Pros: familiar with the language; Cons: have to use scripts to do everything.
My day-to-day activities envolves but not limit to :
Checking the text logs and/or xml.
Grepping important sections from logs for further analysis.
some automate process like application server configuration etc...
automated functional testing - and result comparison
some system admin's job, networking diagnostics, checking process and services, etc...
Any good ideas? Thanks a lot in advance!
While I am a die-hard Linux fan I would recommend in your case to look at Cygwin, preferably on a USB drive or similar. It can live in a single directory, be started with a simple script and end up with (almost) all the Unix goodness, but still being able to access all of the host platform resources.
There are the usual warts related to / vs \ and even worse the case insensitive but case preserving filenames with lot's of spaces in them, but that's equally obnoxious on any other command line.
There is also Mingw but it's scope is more limited I found. It works exceedingly well in a couple of selected target areas, but less so for a GP wide unix-like environment.
I have had a cygwin folder on all my windows machines (and the ones I had to use/repair/maintain) for a very long time now.
My objective is create an apache module that will provide RESTful services (i.e., we have some legacy code that controls/queries some networking equipment and we would now like to expose that functionality as a RESTful service). I guess the flow might look something like this:
WebBrowser -- issues RESTful URI---> [Apache (my_module) ] -->..
..---> Interface to existing Legacy code.
I have been mucking around various wikis, blogs, forums, articles etc. but I just can't seem to understand how those RESTful urls will get to (my_module) in apache [you can tell I have never worked with web-servers internals, much less modules, before]. I mean, do I have to edit that httpd.conf file and say something like: Send all urls that look like http://baseurl/restservices/... to my_module. If so, how do I do it?
Also, what will my_module actually get? Does it get the full http request message and it has to parse it like typical CGI programs?
Further, what is the best way for my_module to interact with my legacy code? E.g., Open a TCP connection to it and send messages and write wrapper around legacy code to interpret those messages. Or can my_module directly invoke the functions in my legacy code somehow if I compiled my entire legacy code as a module in apache?
Thanks for any hints. If u know of a good tutorial, please point me to it. I'm looking for a high level overview that will give me the architecture (the developers under me can then follow up on the nitty-gritty details).
I'd write an extension for PHP or Python and use mod_php / mod_wsgi
I think you are approaching this in the wrong way:
Apache modules are not really how you want to handle a URL if your requirements are quote basic. Depending on the language your legacy code is in, I would advise:
Binding its API into a python or PHP module, and have that script called by Apache through normal means. It is also a lot simple (in many cases) to glue a C-call style compiled language to these scripting languages rather than Apache itself.
It also has the advantage of adding an abstractions which allows you to layer additional logic in a scripting language on your core legacy code. You may also want to preprocess data and validate it from the request before handing it into your legacy code.
Both PHP and Python also have RESTful frameworks and utilities.
If you do write an Apache module, then check out Writing Apache Modules with Perl and C
See:
Developing PHP Extensions in C, Extending Python in C or C++ ... also if using Python checkout the WSGI stuff.
I'd agree with Aiden. Writing Apache modules is not for the faint hearted and you definitely don't want to go there unless you absolutely must. You would need to be prepared to become very conversant with how Apache works.
If you still think you need to, then look at:
http://httpd.apache.org/apreq/
This is a library which uses existing Apache Runtime Libraries and which provides higher level functionality for dealing with POST data, cookies etc from C code hooked into Apache via a custom module.
The book Aiden mentions though is a bit dated. Better off getting:
The Apache Modules Book: Application Development with Apache