There are plenty examples using Selenium Python to handle shadow DOM. I'd like to do the same in Perl.
Perl's Selenium::Remote::Driver doesn't have shadow DOM support, but I should be able to do it through JavaScript. I got my inspiration from
accessing-shadow-dom-tree-with-selenium.
The following is my code in Perl
#!/usr/bin/env perl
use Selenium::Chrome;
my $driver = Selenium::Chrome->new (
startup_timeout => 60,
custom_args => "--log-path=/tmp/selenium_chromedriver",
logfile => "/tmp/selenium_chromedriver2",
debug_on => 1,
extra_capabilities => {
'goog:chromeOptions' => {
args => [
'--no-sandbox',
'--disable-dev-shm-usage',
'--window-size=1260,720',
'--user-data-dir=/tmp/selenium_chrome',
],
},
},
);
$driver->get("chrome-search://local-ntp/local-ntp.html"); # chrome new tab
my $shadow_host = $driver->find_element("html/body/ntp-app", "xpath");
my $shadow_root = $driver->execute_script('return arguments[0].shadowRoot', $shadow_host);
for my $e ( #{$shadow_root->find_elements(':host > *', 'css')} ) {
# error: Can't call method "find_elements" on unblessed reference
print "found\n";
}
$driver->shutdown_binary();
But I got error: Can't call method "find_elements" on unblessed reference.
How can I overcome this error?
Thank you for any help.
My environment is: ubuntu 18, Perl 5.26, Selenium:Chrome 1.46, Chrome
99, chromedriver 99.
The same mechanism is tested working with Python
3.8.5.
Why I am not using Python? because the server in my work place only has Perl, no Python 3.
the following code works
#!/usr/bin/env perl
use Selenium::Chrome;
my $driver = Selenium::Chrome->new (
startup_timeout => 60,
custom_args => "--log-path=/tmp/selenium_chromedriver",
logfile => "/tmp/selenium_chromedriver2",
debug_on => 1,
extra_capabilities => {
'goog:chromeOptions' => {
args => [
'--no-sandbox',
'--disable-dev-shm-usage',
'--window-size=1260,720',
'--user-data-dir=/tmp/selenium_chrome',
],
},
},
);
$driver->get("chrome-search://local-ntp/local-ntp.html"); # chrome new tab
my $shadow_host = $driver->find_element("html/body/ntp-app", "xpath");
package MyShadow {
sub new {
my ($class, %attrs) = #_;
my $shadow_root = $attrs{driver}->execute_script('return arguments[0].shadowRoot', $attrs{shadow_host});
return undef if ! $shadow_root;
$attrs{shadow_root} = $shadow_root;
bless \%attrs, $class;
}
sub find_element {
my ($self, $target, $scheme) = #_;
die "scheme=$scheme is not supported. Only css is supported" if $scheme ne 'css';
return $self->{driver}->execute_script(
"return arguments[0].querySelector(arguments[1])",
$self->{shadow_root},
$target
);
}
sub find_elements {
my ($self, $target, $scheme) = #_;
die "scheme=$scheme is not supported. Only css is supported" if $scheme ne 'css';
return $self->{driver}->execute_script(
"return arguments[0].querySelectorAll(arguments[1])",
$self->{shadow_root},
$target
);
}
};
my $shadow_driver = MyShadow->new(driver=>$driver, shadow_host=>$shadow_host);
if ($shadow_driver) {
for my $e ( #{$shadow_driver->find_elements(':host > *', 'css')} ) {
print "found\n";
}
}
$driver->shutdown_binary();
Key points:
For Selenium, no matter Python or Perl, they are wrappers to
javascript. As long as you get correct javascript, you can do
whatever you want.
For Shadow driver, all you need to implement is the find_element() and find_elements().
I only implemented 'css', no 'xpath', because that's what Python does as of 2022/09/19.
Related
I'm attempting to run a series of shell commands in parallel in Perl6, using Perl5's Parallel::ForkManager
This is an almost exact translation of working Perl5 code.
CONTROL {
when CX::Warn {
note $_;
exit 1;
}
}
use fatal;
role KeyRequired {
method AT-KEY (\key) {
die "Key {key} not found" unless self.EXISTS-KEY(key);
nextsame;
}
}
use Parallel::ForkManager:from<Perl5>;
sub run_parallel (#cmd) {
my $manager = Parallel::ForkManager(8).new();
for (#cmd) -> $command {
$manager.start and $manager.next;
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
$manager.finish;
}
$manager.wait_all_children;#necessary after all lists
}
my #cmd;
my Str $dir = 'A/1';
for dir($dir, test => /\.vcf\.gz$/) -> $vcf {
#cmd.append: "aws s3 cp $vcf s3://s3dir/$dir/"
}
put #cmd.elems;
run_parallel(#cmd);
Basically, I'm trying to parallelize tedious shell commands.
However, this mysterious error comes up:
Cannot invoke this object (REPR: P6opaque; Parallel::ForkManager) in
sub run_parallel at 2.aws_cp.p6 line 18 in block at
2.aws_cp.p6 line 39
Why is Perl6 saying this? what is wrong? how can I get these commands to run?
Perhaps there is a more native/idiomatic way to run shell commands in parallel in Perl6?
You probably want to look at using Proc::Async which runs external commands asynchronously in threads without forking separate instances of the code to do it.
Perl5's Parallel::ForkManager probably won't work in Perl6 because of how Inline::Perl5 is implemented.
Inline::Perl5 embeds the Perl5 compiler/runtime inside of Perl6.
Parallel::ForkManager expects that Perl5 was run by itself.
If you ever did get it to do something other than generate an error it would probably screw up the Perl6 runtime. The main problem is the use of fork. For more information about why fork is a a problem see the article Bart Wiegmans (brrt) wrote about it: “A future for fork(2)”
Perl6 already has a similar feature that is easier to use.
sub run_parallel (#cmd) {
my #children = do for (#cmd) -> $command {
start {
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
}
}
await #children;
}
start is a prefix that tells the runtime to start running the following code sometime in the near future. It returns a Promise.
await takes a list of Promises and returns a list of their results.
start basically calls Promise.start which is similar to:
sub start ( &code ) {
my $promise = Promise.new;
my $vow = $promise.vow;
$*SCHEDULER.cue(
{ $vow.keep(code(|c)) },
:catch(-> $ex { $vow.break($ex); }) );
$promise
}
So it will use the globally available thread pool in $*SCHEDULER. If you want to use a separate one you could.
sub run_parallel (#cmd) {
my $*SCHEDULER = ThreadPoolScheduler.new(max_threads => 8);
my #children = do for (#cmd) -> $command {
start {
my $proc = shell $command, :out, :err;
if $proc.exitcode != 0 {
put "$command failed";
put $proc.out.slurp;
put $proc.err.slurp;
die;
}
}
}
await #children;
}
It would make more sense to use Proc::Async for this though.
Is it possible in Perl6 to find all installed modules whose file-name matches a pattern?
In Perl5 I would write it like this:
use File::Spec::Functions qw( catfile );
my %installed;
for my $dir ( #INC ) {
my $glob_pattern = catfile $dir, 'App', 'DBBrowser', 'DB', '*.pm';
map { $installed{$_}++ } glob $glob_pattern;
}
There is currently no way to get the original file name of an installed module. However it is possible to get the module names
sub list-installed {
my #curs = $*REPO.repo-chain.grep(*.?prefix.?e);
my #repo-dirs = #curs>>.prefix;
my #dist-dirs = |#repo-dirs.map(*.child('dist')).grep(*.e);
my #dist-files = |#dist-dirs.map(*.IO.dir.grep(*.IO.f).Slip);
my $dists := gather for #dist-files -> $file {
if try { Distribution.new( |%(from-json($file.IO.slurp)) ) } -> $dist {
my $cur = #curs.first: {.prefix eq $file.parent.parent}
take $_ for $dist.hash<provides>.keys;
}
}
}
.say for list-installed();
see: Zef::Client.list-installed()
I'm looking to toggle IE8 mode in my LESS files and automated the file generation in Gulp.
This is where I stopped in what to pass gulp-less (minus a bunch of stuff):
var IE = true;
var LESSConfig = {
plugins: [ ... ],
paths: LESSpath,
ie8compat: IE, //may as well toggle this
// Set in variables.less, #ie:false; - used in mixin & CSS guards
// many variations tried
// globalVars: [ { "ie":IE } ],
modifyVars:{ "ie":IE }
};
...
.pipe( less ( LESSConfig ) )
Is variable modification not supported in Gulp?
I'd like to avoid using gulp-modify et al if I can. I'd like to keep the build system fairly abstracted from the source files.
modifyVars is working for me now:
...
var LESSConfig = {
paths: paths.LESSImportPaths,
plugins: [
LESSGroupMediaQueries,
LESSautoprefix
],
modifyVars: {
ie: 'false'
}
};
var LESSConfigIE = {
paths: paths.LESSImportPaths,
modifyVars: {
ie: 'true'
}
};
function processLESS (src, IE, dest){
return gulp.src(src)
.pipe( $.if( IE, $.less( LESSConfigIE ), $.less( LESSConfig ) ) )
.pipe( $.if( IE, $.rename(function(path) { path.basename += "-ie"; }) ) )
.pipe( gulp.dest(dest) )
}
// build base.css files
gulp.task('base', function() {
return processLESS( paths.Base + '/*.less', false, paths.dest );
});
// build base-ie.css files for IE
gulp.task('baseIE', function() {
return processLESS( paths.Base + '/*.less', true, paths.dest );
});
Since I could not get this to work with gulp-lessand it became apparent to me that the application of globalVars and modifyVars are both broken, I came up with a different solution.
You can use gulp-append-prepend to write your variables into the file before gulp-less processes it. A little less elegant but, on the plus side, it actually works.
Something like this:
gulp.src('main.less')
.pipe(gap.prependText('#some-global-var: "foo";'))
.pipe(gap.appendText('#some-modify-var: "bar";'))
.pipe(less())
.pipe(gulp.dest('./dest/'));
Nowadays (2019) this problem seems to be fixed.
However it cost me still a lot of time to get it running.
Here is what I did:
gulp.task('lessVariants', ['less'], function() {
return gulp.src('less/styles.less', {base:'less/'})
.pipe(less({modifyVars:{'#color1': '#535859'}))
.pipe(less({modifyVars:{'#color2': '#ff0000'}))
.pipe(less({modifyVars:{'#color3': '#ccffcc'}))
.pipe(rename('styles.modified.css'))
.pipe(cleanCSS())
.pipe(gulp.dest(distFolder + 'css'))
})
This did not work. Only the last variable was modified. I changed it as follows to get it working:
gulp.task('lessVariants', ['less'], function() {
return gulp.src('less/styles.less', {base:'less/'})
.pipe(less({modifyVars: {
'#color1': '#535859',
'#color2': '#ff0000',
'#color3': '#ccffcc',
}}))
.pipe(rename('styles.variant.css'))
.pipe(cleanCSS())
.pipe(gulp.dest(distFolder + 'css'))
})
Im using Postgres 9.3 on MacOSX.
I would be very happy if anyone could point me in the right direction here. I would like to write a function which connects to an existing (online) database (e.g. this one) and retrieves data (in this case shapefiles) using an input file with appropriate strings (in this case MRGIDs). Im sorry I don't have any code, I literally don't know where to start and I don't seem to find any threads on it. Maybe SQL isn't the way to go here?
Input file example;
species,mrgids
Sp1,4279
Sp1,8366
Sp1,21899
Sp1,21834
Sp1,7130
Sp1,1905
Sp1,21900
Sp1,5679
Sp1,5696
Thanks!
This is almost certainly done best outside the database, using a script in your choice of language. I'd use Python and psycopg2, but things like Ruby + the Pg gem, Perl and DBI / DBD::Pg, or even PHP and PDO, are equally reasonable choices.
Your code can do an HTTP fetch, then (if it's CSV-like) use the COPY command to load the data into PostgreSQL. If it's a shapefile, you can feed the data to PostGIS's shp2pgsql loader, or make individual INSERTs using the GeomFromText PostGIS function.
You could do the HTTP fetch from a PL/Pythonu or PL/Perlu stored procedure, but there'd be no real benefit over just doing it from a script, and it'd be more robust as an external script.
So, really, you need to split this problem up.
You'll need code to talk to the website(s) of interest, possibly including things like HTTP POSTs to submit forms. Or, preferably, use a web API for the site(s) that's designed for automatic scripted interaction. Most simple RESTful APIs are easy to use from scripting languages using libraries like Perl's LWP, Python's httplib, etc. In the case of the site you linked to, as user623952 mentioned, there's a RESTful API.
Then you'll need code to fetch the data of interest, and code to read the fetched data and load it into PostgreSQL. You might want to download all the data then load it, or you may want to stream it into the database as it's downloaded (pipe to shp2pgsql, etc).
this a very basic example with with PHP and CURL
I used your input file exactly and saved it as input.txt
species,mrgids
Sp1,4279
Sp1,8366
Sp1,21899
Sp1,21834
Sp1,7130
Sp1,1905
Sp1,21900
Sp1,5679
Sp1,5696
and this is the PHP and CURL doing its stuff:
<?php
$base_url = "http://www.marineregions.org/rest/getGazetteerRecordByMRGID.json/%s/";
// just get the input file into an array to use
$csv = read_file("input.txt");
// if you want to see the format of $csv
print "<pre>".print_r($csv,true)."</pre>";
// go through each csv item and run a curl request on it
foreach($csv as $i => $data)
{
$mrgids = $data['mrgids'];
$url = sprintf($base_url,$mrgids);
$response = run_curl_request($url);
if ($response!==false)
{
//http://us2.php.net/manual/en/function.json-decode.php
$json = json_decode($response,true);
if (!is_null($json))
{
// this is where you would write the code to stick this info in
// your DB or do whatever you want with it...
print "<pre>$url \n".print_r($json,true)."\n\n</pre>";
}
else
{
print "error: response was not proper JSON for $url <br/><br/>";
print $response."<br/><br/><br/>";
}
}
else
{
print "error: response was false for $url <br/><br/>";
}
}
function read_file($filename, $has_headers=true, $assoc=true)
{
$headers = array();
$row = 1;
if (($handle = fopen($filename, "r")) !== FALSE)
{
$return = array();
if ($has_headers)
{
if (($data = fgetcsv($handle, 1000, ",")) !==false)
{
$headers = $data;
}
}
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
if ($assoc)
{
$temp = array();
foreach($headers as $hi => $header)
{
$temp[$header] = (isset($data[$hi])) ? $data[$hi] : '';
}
$return[] = $temp;
}
else
{
$return[] = $data;
}
}
fclose($handle);
}
else
{
$return = false;
}
return $return;
}
// requires PHP CURL extension
// http://php.net/manual/en/function.curl-setopt.php
function run_curl_request($url)
{
// create curl resource
$ch = curl_init();
$defaults = array(
CURLOPT_POST => false,
CURLOPT_HEADER => false,
CURLOPT_URL => $url,
CURLOPT_FRESH_CONNECT => true,
CURLOPT_FAILONERROR => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FORBID_REUSE => true,
CURLOPT_TIMEOUT => 4
);
curl_setopt_array($ch, $defaults);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
return $output;
}
?>
And if it worked, you get a bunch of these as output:
http://www.marineregions.org/rest/getGazetteerRecordByMRGID.json/4279/
Array
(
[MRGID] => 4279
[gazetteerSource] => IHO 23-3rd: Limits of Oceans and Seas, Special Publication 23, 3rd Edition 1953, published by the International Hydrographic Organization.
[placeType] => IHO Sea Area
[latitude] => 39.749996185303
[longitude] => 5.0942182540894
[minLatitude] => 35.071937561035
[minLongitude] => -6.0326728820801
[maxLatitude] => 44.42805480957
[maxLongitude] => 16.221109390259
[precision] => 1079464.0796258
[preferredGazetteerName] => Mediterranean Sea - Western Basin
[preferredGazetteerNameLang] => English
[status] => standard
[accepted] => 4279
)
notes:
I had to do this to get CURL to work on WAMP for PHP 5.3.13
json_decde()
curl_setopt()
curl_exec()
fgetcsv()
curl_multi_exec() - look into this if you chose this route, you will want it
i'm programming a mod activity for moodle which load files and show'em to any student who can access to the course.
The problem is that handing files in moodle is damn hard.
this is what i have done so far:
option page with importers
$mform->addElement('filepicker', 'slidesyncmedia', get_string('slidesyncmedia', 'slidesync'), null, array('maxbytes' => $maxbytes, 'accepted_types' => '*'));
$mform->addElement('filemanager', 'slidesyncslides', get_string('slidesyncslides', 'slidesync'), null, array('subdirs' => 0, 'maxbytes' => $maxbytes, 'maxfiles' => 50, 'accepted_types' => array('*') ));
after submit the files are stored in draft
and everything is loaded in another page that save all on db
if ($draftitemid = file_get_submitted_draft_itemid('slidesyncmedia')) {
file_save_draft_area_files($draftitemid, $context->id, 'mod_slidesync', 'slidesyncmedia', 0, array('subdirs' => 0, 'maxfiles' => 1));
}
if ($draftitemid = file_get_submitted_draft_itemid('slidesyncslides')) {
file_save_draft_area_files($draftitemid, $context->id, 'mod_slidesync', 'slidesyncslides', 0, array('subdirs' => 0, 'maxfiles' => 50));
}
in the end i use again the first page in another place (if files are there, then shows them)
$fs = get_file_storage();
if ($files = $fs->get_area_files($context->id, 'mod_slidesync', 'slidesyncslides', '0', 'sortorder', false)) {
// Look through each file being managed
foreach ($files as $file) {
// Build the File URL. Long process! But extremely accurate.
$fileurl = moodle_url::make_pluginfile_url($file->get_contextid(), $file->get_component(), $file->get_filearea(), $file->get_itemid(), $file->get_filepath(), $file->get_filename());
echo $fileurl;
}
} else {
echo '<p>Please upload an image first</p>';
}
this make an url but if clicked moodle says that file does not exist
mysite.com/pluginfile.php/53/mod_slidesync/slidesyncslides/0/Koala.jpg
in the db the file are correctly saved!!!
53 mod_slidesync slidesyncslides 0 / Koala.jpg
what i'm missing?
thanks
A long time passed but I was working on a plugin and had the same problem.
I manage to solve it.
To provide the file you need to create the:
function MYPLUGIN_pluginfile($course, $cm, $context, $filearea, $args, $forcedownload, array $options=array())
Here is the example function: https://docs.moodle.org/dev/File_API#Serving_files_to_users
Remember change the last call to send_file to send_stored_file in Moodle 2.3+