How can I do a SQL query to an Oracle database with Perl and get the result as JSON? - sql

I'm working with a legacy system and need to get data out of an Oracle database using Perl. Perl is one of the languages I don't spend much time in, so I'd like to be able to run a simple SQL query and pass the data to another system via JSON.
It seems that JSON, DBI, and DBD are available on this system. I'd like to accomplish this without making too many changes or updates to the system, if possible. I believe the JSON library is at version 5.12.2
I found DBI-Link library on Github and I believe this file is almost exactly what I need:
#!/usr/bin/perl -l
use strict;
use warnings;
$|++;
use JSON;
use DBI;
use DBD::Oracle qw(:ora_types);
my $dbh = DBI->connect(
'dbi:Oracle:host=localhost;sid=xe',
'hr',
'foobar',
{
AutoCommit => 1,
RaiseError => 1,
}
);
my #methods = qw(table_info column_info primary_key_info);
foreach my $method (#methods) {
if ( $dbh->can($method) ) {
print "Handle has method $method. w00t!"
}
else {
$dbh->disconnect;
print "Sadly, handle does not have method $method. D'oh!";
exit;
}
}
my $sth=$dbh->table_info('%', '%', '%', 'TABLE');
while(my $table = $sth->fetchrow_hashref) {
my $t;
$t->{'Table Name'} = $table->{TABLE_NAME};
$t->{'Column Info'} = $dbh->column_info(
undef,
$table->{TABLE_SCHEM},
$table->{TABLE_NAME},
'%'
)->fetchall_arrayref({});
$t->{'Primary Key Info'} = $dbh->primary_key_info(
undef,
$table->{TABLE_SCHEM},
$table->{TABLE_NAME}
)->fetchall_arrayref({});
print map {"$_: ". json_encode($t->{$_})} grep{ defined $t->{$_} } 'Table Name', 'Column Info', 'Primary Key Info';
print;
}
$sth->finish;
$dbh->disconnect;
The Error
I've installed the dependencies but when I run it I am getting:
Undefined subroutine &main::json_encode called at ./oracle.t line 47.
I searched the rest of the source in that repository and don't see any my json_encode definition, so maybe I have a version of the JSON library that is too old is my possible idea, but it seems unlikely that the json_encode method would have changed names.
The Next Steps
After I get json_encode to work I know I will need to execute a custom query and then save the data, it would be something like this:
$sth = $dbh->prepare("select * from records where pending = 1");
$sth->execute;
my $records = new HASH;
while($r = $sth->fetchrow_hashref)
{
$records << $r
}
my $json = json_encode($records)
However I'm unsure how to build the $records object for encoding so any help would be appreciated. I have searched stackoverflow, google, and github for perl examples of oracle to json and only had luck with the code from that DBI-Link repo.

According to the documentation for the JSON module, the function you want is encode_json and not json_encode.
I'd probably store the records in an array of hashes; something like this:
my #records;
while (my $r = $sth->fetchrow_hashref)
{
push(#records, $r);
}
If you know what field you want a hash-of-hashes keyed on:
my %records;
while (my $r = $sth->fetchrow_hashref)
{
$records{ $r->{key_field} } = $r;
}

Related

Perl6: Cannot invoke this object (REPR: P6opaque; Parallel::ForkManager)

I'm attempting to run a series of shell commands in parallel in Perl6, using Perl5's Parallel::ForkManager
This is an almost exact translation of working Perl5 code.
CONTROL {
when CX::Warn {
note $_;
exit 1;
}
}
use fatal;
role KeyRequired {
method AT-KEY (\key) {
die "Key {key} not found" unless self.EXISTS-KEY(key);
nextsame;
}
}
use Parallel::ForkManager:from<Perl5>;
sub run_parallel (#cmd) {
my $manager = Parallel::ForkManager(8).new();
for (#cmd) -> $command {
$manager.start and $manager.next;
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
$manager.finish;
}
$manager.wait_all_children;#necessary after all lists
}
my #cmd;
my Str $dir = 'A/1';
for dir($dir, test => /\.vcf\.gz$/) -> $vcf {
#cmd.append: "aws s3 cp $vcf s3://s3dir/$dir/"
}
put #cmd.elems;
run_parallel(#cmd);
Basically, I'm trying to parallelize tedious shell commands.
However, this mysterious error comes up:
Cannot invoke this object (REPR: P6opaque; Parallel::ForkManager) in
sub run_parallel at 2.aws_cp.p6 line 18 in block at
2.aws_cp.p6 line 39
Why is Perl6 saying this? what is wrong? how can I get these commands to run?
Perhaps there is a more native/idiomatic way to run shell commands in parallel in Perl6?
You probably want to look at using Proc::Async which runs external commands asynchronously in threads without forking separate instances of the code to do it.
Perl5's Parallel::ForkManager probably won't work in Perl6 because of how Inline::Perl5 is implemented.
Inline::Perl5 embeds the Perl5 compiler/runtime inside of Perl6.
Parallel::ForkManager expects that Perl5 was run by itself.
If you ever did get it to do something other than generate an error it would probably screw up the Perl6 runtime. The main problem is the use of fork. For more information about why fork is a a problem see the article Bart Wiegmans (brrt) wrote about it: “A future for fork(2)”
Perl6 already has a similar feature that is easier to use.
sub run_parallel (#cmd) {
my #children = do for (#cmd) -> $command {
start {
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
}
}
await #children;
}
start is a prefix that tells the runtime to start running the following code sometime in the near future. It returns a Promise.
await takes a list of Promises and returns a list of their results.
start basically calls Promise.start which is similar to:
sub start ( &code ) {
my $promise = Promise.new;
my $vow = $promise.vow;
$*SCHEDULER.cue(
{ $vow.keep(code(|c)) },
:catch(-> $ex { $vow.break($ex); }) );
$promise
}
So it will use the globally available thread pool in $*SCHEDULER. If you want to use a separate one you could.
sub run_parallel (#cmd) {
my $*SCHEDULER = ThreadPoolScheduler.new(max_threads => 8);
my #children = do for (#cmd) -> $command {
start {
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
}
}
await #children;
}
It would make more sense to use Proc::Async for this though.

Perl6: large gzipped files read line by line

I'm trying to read a gz file line by line in Perl6, however, I'm getting blocked:
How to read gz file line by line in Perl6 however, this method, reading everything into :out uses far too much RAM to be usable except on very small files.
I don't understand how to use Perl6's Compress::Zlib to get everything line by line, although I opened an issue on their github https://github.com/retupmoca/P6-Compress-Zlib/issues/17
I'm trying Perl5's Compress::Zlib to translate this code, which works perfectly in Perl5:
use Compress::Zlib;
my $file = "data.txt.gz";
my $gz = gzopen($file, "rb") or die "Error reading $file: $gzerrno";
while ($gz->gzreadline($_) > 0) {
# Process the line read in $_
}
die "Error reading $file: $gzerrno" if $gzerrno != Z_STREAM_END ;
$gz->gzclose() ;
to something like this using Inline::Perl5 in Perl6:
use Compress::Zlib:from<Perl5>;
my $file = 'chrMT.1.vcf.gz';
my $gz = Compress::Zlib::new(gzopen($file, 'r');
while ($gz.gzreadline($_) > 0) {
print $_;
}
$gz.gzclose();
but I can't see how to translate this :(
I'm confused by Lib::Archive example https://github.com/frithnanth/perl6-Archive-Libarchive/blob/master/examples/readfile.p6 I don't see how I can get something like item 3 here
There should be something like
for $file.IO.lines(gz) -> $line { or something like that in Perl6, if it exists, I can't find it.
How can I read a large file line by line without reading everything into RAM in Perl6?
Update Now tested, which revealed an error, now fixed.
Solution #2
use Compress::Zlib;
my $file = "data.txt.gz" ;
my $handle = try open $file or die "Error reading $file: $!" ;
my $zwrap = zwrap($handle, :gzip) ;
for $zwrap.lines {
.print
}
CATCH { default { die "Error reading $file: $_" } }
$handle.close ;
I've tested this with a small gzipped text file.
I don't know much about gzip etc. but figured this out based on:
Knowing P6;
Reading Compress::Zlib's README and choosing the zwrap routine;
Looking at the module's source code, in particular the signature of the zwrap routine our sub zwrap ($thing, :$zlib, :$deflate, :$gzip);
And trial and error, mainly to guess that I needed to pass the :gzip adverb.
Please comment on whether my code works for you. I'm guessing the main thing is whether it's fast enough for the large files you have.
A failed attempt at solution #5
With solution #2 working I would have expected to be able to write just:
use Compress::Zlib ;
.print for "data.txt.gz".&zwrap(:gzip).lines ;
But that fails with:
No such method 'eof' for invocant of type 'IO::Path'
This is presumably because this module was written before the reorganization of the IO classes.
That led me to #MattOates' IO::Handle like object with .lines ? issue. I note no response and I saw no related repo at https://github.com/MattOates?tab=repositories.
I am focusing on the Inline::Perl5 solution that you tried.
For the call to $gz.gzreadline($_): it seems like gzreadline tries to return the line read from the zip file by modifying its input argument $_ (treated as an output argument, but it is not a true Perl 5 reference variable[1]), but the modified value is not returned to the Perl 6 script.
Here is a possoble workaround:
Create a wrapper module in the curent directory, e.g. ./MyZlibWrapper.pm:
package MyZlibWrapper;
use strict;
use warnings;
use Compress::Zlib ();
use Exporter qw(import);
our #EXPORT = qw(gzopen);
our $VERSION = 0.01;
sub gzopen {
my ( $fn, $mode ) = #_;
my $gz = Compress::Zlib::gzopen( $fn, $mode );
my $self = {gz => $gz};
return bless $self, __PACKAGE__;
}
sub gzreadline {
my ( $self ) = #_;
my $line = "";
my $res = $self->{gz}->gzreadline($line);
return [$res, $line];
}
sub gzclose {
my ( $self ) = #_;
$self->{gz}->gzclose();
}
1;
Then use Inline::Perl5 on this wrapper module instead of Compress::Zlib. For example ./p.p6:
use v6;
use lib:from<Perl5> '.';
use MyZlibWrapper:from<Perl5>;
my $file = 'data.txt.gz';
my $mode = 'rb';
my $gz = gzopen($file, $mode);
loop {
my ($res, $line) = $gz.gzreadline();
last if $res == 0;
print $line;
}
$gz.gzclose();
[1]
In Perl 5 you can modify an input argument that is not a reference, and the change will be reflected in the caller. This is done by modifying entries in the special #_ array variable. For example: sub quote { $_[0] = "'$_[0]'" } $str = "Hello"; quote($str) will quote $str even if $str is not passed by reference.

Perl web server: How to route

As seen in my code below, I am using apache to serve my Perl web server. I need Perl to have multple routes for my client as seen in my %dispatch. If I figure one out I'm sure the rest will be very similar. If we look at my Subroutine sub resp_index, how can I modify this to link to my index.html file located in my root: /var/www/perl directory?
/var/www/perl/perlServer.pl:
#!/usr/bin/perl
{
package MyWebServer;
use HTTP::Server::Simple::CGI;
use base qw(HTTP::Server::Simple::CGI);
my %dispatch = (
'/index.html' => \&resp_index,
# ...
);
sub handle_request {
my $self = shift;
my $cgi = shift;
my $path = $cgi->path_info();
my $handler = $dispatch{$path};
if (ref($handler) eq "CODE") {
print "HTTP/1.0 200 OK\r\n";
$handler->($cgi);
} else {
print "HTTP/1.0 404 Not found\r\n";
print $cgi->header,
$cgi->start_html('Not found'),
$cgi->h1('Not found'),
$cgi->end_html;
}
}
sub resp_index {
my $cgi = shift; # CGI.pm object
return if !ref $cgi;
my $who = $cgi->param('name');
print $cgi->header,
$cgi->start_html("index"),
$cgi-h1("THIS IS INDEX"),
$cgi->end_html;
}
}
my $pid = MyWebServer->new()->background();
print "Use 'kill $pid' to stop server.\n";
I think what you're asking is how do you serve a file from your web server? Open it and print it, like any other file.
use autodie;
sub resp_index {
my $cgi = shift;
return if !ref $cgi;
print $cgi->header;
open my $fh, "<", "/var/www/perl/index.html";
print <$fh>;
}
Unless this is an exercise, really, really, REALLY don't write your own web framework. It's going to be slow, buggy, and insecure. Consider a small routing framework like Dancer.
For example, mixing documents like index.html and executable code like perlServer.pl in the same directory invites a security hole. Executable code should be isolated in their own directory so they can be given wholly different permissions and stronger protection.
Let's talk about this line...
return if !ref $cgi;
This line is hiding an error. If your functions are passed the wrong argument, or no argument, it will silently return and you (or the person using this) will have no idea why nothing happened. This should be an error...
use Carp;
croak "resp_index() was not given a CGI object" if !ref $cgi;
...but really you should use one of the existing function signature modules such as Method::Signatures.
use Method::Signatures;
func resp_index(CGI $cgi) {
...
}

retrieve data from online database using input file

Im using Postgres 9.3 on MacOSX.
I would be very happy if anyone could point me in the right direction here. I would like to write a function which connects to an existing (online) database (e.g. this one) and retrieves data (in this case shapefiles) using an input file with appropriate strings (in this case MRGIDs). Im sorry I don't have any code, I literally don't know where to start and I don't seem to find any threads on it. Maybe SQL isn't the way to go here?
Input file example;
species,mrgids
Sp1,4279
Sp1,8366
Sp1,21899
Sp1,21834
Sp1,7130
Sp1,1905
Sp1,21900
Sp1,5679
Sp1,5696
Thanks!
This is almost certainly done best outside the database, using a script in your choice of language. I'd use Python and psycopg2, but things like Ruby + the Pg gem, Perl and DBI / DBD::Pg, or even PHP and PDO, are equally reasonable choices.
Your code can do an HTTP fetch, then (if it's CSV-like) use the COPY command to load the data into PostgreSQL. If it's a shapefile, you can feed the data to PostGIS's shp2pgsql loader, or make individual INSERTs using the GeomFromText PostGIS function.
You could do the HTTP fetch from a PL/Pythonu or PL/Perlu stored procedure, but there'd be no real benefit over just doing it from a script, and it'd be more robust as an external script.
So, really, you need to split this problem up.
You'll need code to talk to the website(s) of interest, possibly including things like HTTP POSTs to submit forms. Or, preferably, use a web API for the site(s) that's designed for automatic scripted interaction. Most simple RESTful APIs are easy to use from scripting languages using libraries like Perl's LWP, Python's httplib, etc. In the case of the site you linked to, as user623952 mentioned, there's a RESTful API.
Then you'll need code to fetch the data of interest, and code to read the fetched data and load it into PostgreSQL. You might want to download all the data then load it, or you may want to stream it into the database as it's downloaded (pipe to shp2pgsql, etc).
this a very basic example with with PHP and CURL
I used your input file exactly and saved it as input.txt
species,mrgids
Sp1,4279
Sp1,8366
Sp1,21899
Sp1,21834
Sp1,7130
Sp1,1905
Sp1,21900
Sp1,5679
Sp1,5696
and this is the PHP and CURL doing its stuff:
<?php
$base_url = "http://www.marineregions.org/rest/getGazetteerRecordByMRGID.json/%s/";
// just get the input file into an array to use
$csv = read_file("input.txt");
// if you want to see the format of $csv
print "<pre>".print_r($csv,true)."</pre>";
// go through each csv item and run a curl request on it
foreach($csv as $i => $data)
{
$mrgids = $data['mrgids'];
$url = sprintf($base_url,$mrgids);
$response = run_curl_request($url);
if ($response!==false)
{
//http://us2.php.net/manual/en/function.json-decode.php
$json = json_decode($response,true);
if (!is_null($json))
{
// this is where you would write the code to stick this info in
// your DB or do whatever you want with it...
print "<pre>$url \n".print_r($json,true)."\n\n</pre>";
}
else
{
print "error: response was not proper JSON for $url <br/><br/>";
print $response."<br/><br/><br/>";
}
}
else
{
print "error: response was false for $url <br/><br/>";
}
}
function read_file($filename, $has_headers=true, $assoc=true)
{
$headers = array();
$row = 1;
if (($handle = fopen($filename, "r")) !== FALSE)
{
$return = array();
if ($has_headers)
{
if (($data = fgetcsv($handle, 1000, ",")) !==false)
{
$headers = $data;
}
}
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
if ($assoc)
{
$temp = array();
foreach($headers as $hi => $header)
{
$temp[$header] = (isset($data[$hi])) ? $data[$hi] : '';
}
$return[] = $temp;
}
else
{
$return[] = $data;
}
}
fclose($handle);
}
else
{
$return = false;
}
return $return;
}
// requires PHP CURL extension
// http://php.net/manual/en/function.curl-setopt.php
function run_curl_request($url)
{
// create curl resource
$ch = curl_init();
$defaults = array(
CURLOPT_POST => false,
CURLOPT_HEADER => false,
CURLOPT_URL => $url,
CURLOPT_FRESH_CONNECT => true,
CURLOPT_FAILONERROR => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FORBID_REUSE => true,
CURLOPT_TIMEOUT => 4
);
curl_setopt_array($ch, $defaults);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
return $output;
}
?>
And if it worked, you get a bunch of these as output:
http://www.marineregions.org/rest/getGazetteerRecordByMRGID.json/4279/
Array
(
[MRGID] => 4279
[gazetteerSource] => IHO 23-3rd: Limits of Oceans and Seas, Special Publication 23, 3rd Edition 1953, published by the International Hydrographic Organization.
[placeType] => IHO Sea Area
[latitude] => 39.749996185303
[longitude] => 5.0942182540894
[minLatitude] => 35.071937561035
[minLongitude] => -6.0326728820801
[maxLatitude] => 44.42805480957
[maxLongitude] => 16.221109390259
[precision] => 1079464.0796258
[preferredGazetteerName] => Mediterranean Sea - Western Basin
[preferredGazetteerNameLang] => English
[status] => standard
[accepted] => 4279
)
notes:
I had to do this to get CURL to work on WAMP for PHP 5.3.13
json_decde()
curl_setopt()
curl_exec()
fgetcsv()
curl_multi_exec() - look into this if you chose this route, you will want it

How can I implement incremental (find-as-you-type) search on command line?

I'd like to write small scripts which feature incremental search (find-as-you-type) on the command line.
Use case: I have my mobile phone connected via USB, Using gammu --sendsms TEXT I can write text messages. I have the phonebook as CSV, and want to search-as-i-type on that.
What's the easiest/best way to do it? It might be in bash/zsh/Perl/Python or any other scripting language.
Edit:
Solution: Modifying Term::Complete did what I want. See below for the answer.
I get the impression GNU Readline supports this kind of thing. Though, I have not used it myself. Here is a C++ example of custom auto complete, which could easily be done in C too. There is also a Python API for readline.
This StackOverflow question gives examples in Python, one of which is ...
import readline
def completer(text, state):
options = [x in addrs where x.startswith(text)]
if state < options.length:
return options[state]
else
return None
readline.set_completer(completer)
this article on Bash autocompletion may help. This article also gives examples of programming bash's auto complete feature.
Following Aiden Bell's hint, I tried Readline in Perl.
Solution 1 using Term::Complete (also used by CPAN, I think):
use Term::Complete;
my $F;
open($F,"<","bin/phonebook.csv");
my #terms = <$F>; chomp(#terms);
close($F);
my $input;
while (!defined $input) {
$input = Complete("Enter a name or number: ",#terms);
my ($name,$number) = split(/\t/,$input);
print("Sending SMS to $name ($number).\n");
system("sudo gammu --sendsms TEXT $number");
}
Press \ to complete, press Ctrl-D to see all possibilities.
Solution 2: Ctrl-D is one keystroke to much, so using standard Term::Readline allows completion and the display off possible completions using only \.
use Term::ReadLine;
my $F;
open($F,"<","bin/phonebook.csv");
my #terms = <$F>; chomp(#terms);
close($F);
my $term = new Term::ReadLine;
$term->Attribs->{completion_function} = sub { return #terms; };
my $prompt = "Enter name or number >> ";
my $OUT = $term->OUT || \*STDOUT;
while ( defined (my $input = $term->readline($prompt)) ) {
my ($name,$number) = split(/\t/,$input);
print("Sending SMS to $name ($number).\n");
system("sudo gammu --sendsms TEXT $number");
}
This solution still needs a for completion.
Edit: Final Solution
Modifying Term::Complete (http://search.cpan.org/~jesse/perl-5.12.0/lib/Term/Complete.pm) does give me on the fly completion.
Source code: http://search.cpan.org/CPAN/authors/id/J/JE/JESSE/perl-5.12.0.tar.gz
Solution number 1 works with this modification. I will put the whole sample online somewhere else if this can be used by somebody
Modifications of Completion.pm (just reusing it's code for Control-D and \ for each character):
170c172,189
my $redo=0;
#match = grep(/^\Q$return/, #cmp_lst);
unless ($#match < 0) {
$l = length($test = shift(#match));
foreach $cmp (#match) {
until (substr($cmp, 0, $l) eq substr($test, 0, $l)) {
$l--;
}
}
print("\a");
print($test = substr($test, $r, $l - $r));
$redo = $l - $r == 0;
if ($redo) { print(join("\r\n", '', grep(/^\Q$return/, #cmp_lst)), "\r\n"); }
$r = length($return .= $test);
}
if ($redo) { redo LOOP; } else { last CASE; }