BigQuery php client library queryresult shows isComplete false. But I can confirm the query is successful - google-bigquery

$queryResults = $this->bigQuery->runQuery($query, ['parameters' => ['id' => $id]]);
$info = $queryResults->info();
var_dump($info);
// var_dump($queryResults);
$isComplete = $queryResults->isComplete();
if ($isComplete) {
exit("Insert. Done!");
}
The query is a "insert select" statement. which I had run successful on bigquery cloud console directly.
When I use php client library here, the query finished successful too and I can confirm that it is the same as I run query directly on google cloud console.
But the info of the query shows isComplete false.
array(3) {
["kind"]=>
string(22) "bigquery#queryResponse"
["jobReference"]=>
array(2) {
["projectId"]=>
string(26) "myprojectid"
["jobId"]=>
string(31) "job_OHcckVSSwAI7pHXijmmUqK5H4XE"
}
["jobComplete"]=>
bool(false)
}
There are no errors report form the $queryResults->info();. How could I find out why it shows me isComplete false?

Probably you still need to run $queryResults -> reload() to update the job status. As in the docs:
$isComplete = $queryResults->isComplete();
while (!$isComplete) {
sleep(1); // small delay between requests
$queryResults->reload();
$isComplete = $queryResults->isComplete();
}

Related

How to execute multiple QueryBuilders in a transaction using TypeORM

I have multiple queries that I created using QueryBuilder, I wanted to execute all of these queries inside a single transaction.
I read the documentation but it lacks information on how to use a QueryBuilder with transactions. I tried to create my queries using the same QueryRunner manager but I really feel like this does nothing.
I also tried to wrap my method using a #Transaction decoration but I get a socket hang up error.
This is my current attempt, I only added 2 queries in the example but I have 5 in total.
let user, membersInstituition
const connection = getConnection()
const queryRunner = connection.createQueryRunner()
console.log(queryRunner.isTransactionActive) // false
await queryRunner.startTransaction()
try {
console.log(queryRunner.isTransactionActive) // true
user = await queryRunner.manager
.createQueryBuilder(Usuarios, 'user')
.leftJoin(UltimaAtualizacaoMobile, 'att', 'user.id_usuario = att.id_usuario')
.select(['user.id_usuario as idUser, user.nome as name, att.ultima_atualizacao as ultimaAtualizacaoMobile'])
.where('user.id_usuario = :idUsuario', { idUsuario: idUsuario })
.execute()
membersInstituition = await queryRunner.manager
.createQueryBuilder(Instituicoes, 'insti')
.leftJoin(Amostras, 'am', 'am.id_instituicao = insti.id_instituicao')
.leftJoin(MembrosInstituicao, 'mi', 'mi.id_instituicao = insti.id_instituicao')
.leftJoin(Permissoes, 'perm', 'perm.id_permissao = mi.id_permissao')
.leftJoin(MembrosAmostra, 'mam', 'mam.id_amostra = am.id_instituicao AND mam.id_membro_instituicao = mi.id_membro_instituicao')
.where('am.id_amostra IN (:...idAmostras)', { idAmostras: idAmostras })
.andWhere('mi.id_usuario = :idUsuario', { idUsuario: idUsuario })
.select('mi.id_membro_instituicao as idMember, mam.id_membro_amostra as idMemberAmostra, insti.id_instituicao as idInstituition, perm.permissao as userPermission')
.orderBy('insti.id_instituicao', 'ASC')
.execute()
await queryRunner.commitTransaction()
console.log(queryRunner.isTransactionActive) // false
} catch (error) {
// Handle error
} finally {
await queryRunner.release()
}
Above code is indeed working, confirmed using TypeORM's logging, it shows the transaction starting and all the following queries before committing.
More info on how to enable Logging and to verify if your queries are under the same transaction: https://github.com/typeorm/typeorm/blob/master/docs/logging.md

Paypal Php Sdk - NotifyUrl is not a fully qualified URL Error

I have this code
$product_info = array();
if(isset($cms['site']['url_data']['product_id'])){
$product_info = $cms['class']['product']->get($cms['site']['url_data']['product_id']);
}
if(!isset($product_info['id'])){
/*
echo 'No product info.';
exit();
*/
header_url(SITE_URL.'?subpage=user_subscription#xl_xr_page_my%20account');
}
$fee = $product_info['yearly_price_end'] / 100 * $product_info['fee'];
$yearly_price_end = $product_info['yearly_price_end'] + $fee;
$fee = ($product_info['setup_price_end'] / 100) * $product_info['fee'];
$setup_price_end = $product_info['setup_price_end'] + $fee;
if(isset($_SESSION['discountcode_amount'])){
$setup_price_end = $setup_price_end - $_SESSION['discountcode_amount'];
unset($_SESSION['discountcode_amount']);
}
$error = false;
$plan_id = '';
$approvalUrl = '';
$ReturnUrl = SITE_URL.'payment/?payment_type=paypal&payment_page=process_agreement';
$CancelUrl = SITE_URL.'payment/?payment_type=paypal&payment_page=cancel_agreement';
$now = $cms['date'];
$now->modify('+5 minutes');
$apiContext = new \PayPal\Rest\ApiContext(
new \PayPal\Auth\OAuthTokenCredential(
$cms['options']['plugin_paypal_clientid'], // ClientID
$cms['options']['plugin_paypal_clientsecret'] // ClientSecret
)
);
use PayPal\Api\ChargeModel;
use PayPal\Api\Currency;
use PayPal\Api\MerchantPreferences;
use PayPal\Api\PaymentDefinition;
use PayPal\Api\Plan;
use PayPal\Api\Patch;
use PayPal\Api\PatchRequest;
use PayPal\Common\PayPalModel;
use PayPal\Api\Agreement;
use PayPal\Api\Payer;
use PayPal\Api\ShippingAddress;
// Create a new instance of Plan object
$plan = new Plan();
// # Basic Information
// Fill up the basic information that is required for the plan
$plan->setName($product_info['name'])
->setDescription($product_info['desc_text'])
->setType('fixed');
// # Payment definitions for this billing plan.
$paymentDefinition = new PaymentDefinition();
// The possible values for such setters are mentioned in the setter method documentation.
// Just open the class file. e.g. lib/PayPal/Api/PaymentDefinition.php and look for setFrequency method.
// You should be able to see the acceptable values in the comments.
$setFrequency = 'Year';
//$setFrequency = 'Day';
$paymentDefinition->setName('Regular Payments')
->setType('REGULAR')
->setFrequency($setFrequency)
->setFrequencyInterval("1")
->setCycles("999")
->setAmount(new Currency(array('value' => $yearly_price_end, 'currency' => $cms['session']['client']['currency']['iso_code'])));
// Charge Models
$chargeModel = new ChargeModel();
$chargeModel->setType('SHIPPING')
->setAmount(new Currency(array('value' => 0, 'currency' => $cms['session']['client']['currency']['iso_code'])));
$paymentDefinition->setChargeModels(array($chargeModel));
$merchantPreferences = new MerchantPreferences();
// ReturnURL and CancelURL are not required and used when creating billing agreement with payment_method as "credit_card".
// However, it is generally a good idea to set these values, in case you plan to create billing agreements which accepts "paypal" as payment_method.
// This will keep your plan compatible with both the possible scenarios on how it is being used in agreement.
$merchantPreferences->setReturnUrl($ReturnUrl)
->setCancelUrl($CancelUrl)
->setAutoBillAmount("yes")
->setInitialFailAmountAction("CONTINUE")
->setMaxFailAttempts("0")
->setSetupFee(new Currency(array('value' => $setup_price_end, 'currency' => $cms['session']['client']['currency']['iso_code'])));
$plan->setPaymentDefinitions(array($paymentDefinition));
$plan->setMerchantPreferences($merchantPreferences);
// ### Create Plan
try {
$output = $plan->create($apiContext);
} catch (Exception $ex){
die($ex);
}
echo $output->getId().'<br />';
echo $output.'<br />';
Been working with paypal php sdk for some days now and my code stop working.
So i went back to basic and i am still getting the same damn error.
I am trying to create a plan for subscription but getting the following error:
"NotifyUrl is not a fully qualified URL"
I have no idea how to fix this as i dont use NotfifyUrl in my code?
Could be really nice if anyone had an idea how to fix this problem :)
Thanks
PayPal did a update to their API last night which has caused problem within their SDK.
They are sending back null values in their responses.
I MUST stress the error is not on sending the request to PayPal, but on processing their response.
BUG Report : https://github.com/paypal/PayPal-PHP-SDK/issues/1151
Pull Request : https://github.com/paypal/PayPal-PHP-SDK/pull/1152
Hope this helps, but their current SDK is throwing exceptions.
Use below simple fix.
Replace below function in vendor\paypal\rest-api-sdk-php\lib\PayPal\Api\MerchantPreferences.php
public function setNotifyUrl($notify_url)
{
if(!empty($notify_url)){
UrlValidator::validate($notify_url, "NotifyUrl");
}
$this->notify_url = $notify_url;
return $this;
}
If you get the same error for return_url/cancel_url, add the if condition as above.
Note: This is not a permanent solution, you can use this until getting the update from PayPal.
From the GitHub repo for the PayPal PHP SDK, I see that the error you mentioned is thrown when MerchantPreferences is not given a valid NotifyUrl. I see you're setting the CancelUrl and ReturnUrl, but not the NotifyUrl. You may simply need to set that as well, i.e.:
$NotifyUrl = (some url goes here)
$obj->setNotifyUrl($NotifyUrl);
Reason behind it!
error comes from.
vendor\paypal\rest-api-sdk-php\lib\PayPal\Validation\UrlValidator.php
line.
if (filter_var($url, FILTER_VALIDATE_URL) === false) {
throw new \InvalidArgumentException("$urlName is not a fully qualified URL");
}
FILTER_VALIDATE_URL: according to this php function.
INVALID URL: "http://cat_n.domain.net.in/"; // IT CONTAIN _ UNDERSCORE.
VALID URL: "http://cat-n.domain.net.in/"; it separated with - dash
here you can dump your url.
vendor\paypal\rest-api-sdk-php\lib\PayPal\Validation\UrlValidator.php
public static function validate($url, $urlName = null)
{
var_dump($url);
}
And then check this here: https://www.w3schools.com/PHP/phptryit.asp?filename=tryphp_func_validate_url
you can check here what character will reason for invalid.

export data from bigquery to cloud storage- php client library - there is one extra empty new line in the cloud storage file

I followed this sample
https://cloud.google.com/bigquery/docs/exporting-data
public function exportDailyRecordsToCloudStorage($date, $tableId)
{
$validTableIds = ['table1', 'table2'];
if (!in_array($tableId, $validTableIds))
{
die("Wrong TableId");
}
$date = date("Ymd", date(strtotime($date)));
$datasetId = $date;
$dataset = $this->bigQuery->dataset($datasetId);
$table = $dataset->table($tableId);
// load the storage object
$storage = $this->storage;
$bucketName = 'mybucket';
$objectName = "daily_records/{$tableId}_" . $date;
$destinationObject = $storage->bucket($bucketName)->object($objectName);
// create the import job
$format = 'NEWLINE_DELIMITED_JSON';
$options = ['jobConfig' => ['destinationFormat' => $format]];
$job = $table->export($destinationObject, $options);
// poll the job until it is complete
$backoff = new ExponentialBackoff(10);
$backoff->execute(function () use ($job) {
print('Waiting for job to complete' . PHP_EOL);
$job->reload();
if (!$job->isComplete()) {
//throw new Exception('Job has not yet completed', 500);
}
});
// check if the job has errors
if (isset($job->info()['status']['errorResult'])) {
$error = $job->info()['status']['errorResult']['message'];
printf('Error running job: %s' . PHP_EOL, $error);
} else {
print('Data exported successfully' . PHP_EOL);
}
I have 37670 rows in my table1, and the cloud storage file has 37671 lines.
And I have 388065 my table2, and the cloud storage file has 388066 lines.
The last line in both cloud storage files is empty line.
Is this a Google BigQuery feature improvement request? or I did something wrong in my codes above?
What you described seems like an unexpected outcome. The output file should generally has the same number of lines as the source table.
Your PHP code looks fine and shouldn't be the cause of the issue.
I'm trying reproduce it but unable to. Could you double-check if the last empty line is somehow added by another tool like a text editor or something? How are you counting the lines of the resulting output.
If you have ruled that out and are sure the newline is indeed added by BigQuery export feature, please consider opening a bug using the BigQuery Issue Tracker as suggested by xuejian and include your job ID so that we can investigate further.

Does ServiceStack RedisClient support Sort command?

my sort command is
"SORT hot_ids by no_keys GET # GET msg:->msg GET msg:->count GET msg:*->comments"
it works fine in redis-cli, but it doesn't return data in RedisClient. the result is a byte[][], length of result is correct, but every element of array is null.
the response of redis is
...
$-1
$-1
...
c# code is
data = redis.Sort("hot_ids ", new SortOptions()
{
GetPattern = "# GET msg:*->msg GET msg:*->count GET msg:*->comments",
Skip = skip,
Take = take,
SortPattern = "not-key"
});
Redis Sort is used in IRedisClient.GetSortedItemsFromList, e.g. from RedisClientListTests.cs:
[Test]
public void Can_AddRangeToList_and_GetSortedItems()
{
Redis.PrependRangeToList(ListId, storeMembers);
var members = Redis.GetSortedItemsFromList(ListId,
new SortOptions { SortAlpha = true, SortDesc = true, Skip = 1, Take = 2 });
AssertAreEqual(members,
storeMembers.OrderByDescending(s => s).Skip(1).Take(2).ToList());
}
You can use the MONITOR command in redis-cli to help diagnose and see what requests the ServiceStack Redis client is sending to redis-server.

How can I alert Skype chatrooms with Jenkins build status?

Our company uses Skype for communications, and I'd like to be able to send alerts to Skype chatrooms when a Jenkins build fails (and when it recovers too).
How can I do this?
I've done this using the Skype Public API
What I did was write a Perl script which uses the SkypeAPI CPAN module to handle the communications with Skype. It's a little clunky, as the script needs to run on a desktop which is running Skype. I run it on my own desktop which is always on anyway, but this does mean the bot appears to be 'me' to the rest of our team.
The end result is fantastic - whenever a jenkins build changes state, the bot sends a message to any Skype chats which have registered an interest by typing *alert. Additionally, any developer can see and share the latest build status by typing *jenkins
Step 1 - Extending the SkypeAPI module
Now, the SkypeAPI module is pretty basic. It has a message loop in the listen() method which checks for new events from the Skype client, and sleeps for a moment if there none.
I wanted my script to hook into this loop so that I could have my bot periodically check Jenkins RSS feed, so I made the following modifications to SkypeAPI.pm after I had installed it with the ActiveState package manager:
I declared new property 'idler' along with the existing properties...
__PACKAGE__->mk_accessors(
qw/api handler_list stop_listen idler/
);
I added a method to set an 'idler' callback which the module will call instead of sleeping
sub register_idler {
my $self = shift;
my $ref_sub = shift;
$self->idler($ref_sub);
}
Finally I modified the message loop to call the idler if set
sub listen {
my $self = shift;
my $idler=$self->idler();
$self->stop_listen(0);
while (!$self->stop_listen) {
my $message;
{
lock #message_list;
$message = shift #message_list;
}
if (not defined $message) {
if ($idler)
{
$self->idler->($self);
}
else
{
sleep 0.1;
}
next;
}
for my $id (sort keys %{$self->handler_list}) {
$self->handler_list->{$id}->($self, $message);
}
}
}
Step 2 - write a robot script
Now the module is a little more capable, it's just a matter of writing a script to act as a bot. Here's mine - I've made a few edits from my original as it contained other irrelevant functionality, but it should give you a starting point.
All of the dependant modules can be installed with the ActiveState package manager.
use strict;
use SkypeAPI;
use LWP::Simple;
use Data::Dumper;
use dirtyRSS;
use Time::Local 'timegm';
use Math::Round;
use Storable;
#CHANGE THIS - where to get jenkins status from
my $jenkinsRss='http://username:password#jenkins.example.com/rssLatest';
my %commands=(
'jenkins' =>\&cmdJenkins,
'alert' =>\&cmdAlert,
'noalert' =>\&cmdNoAlert,
'help' =>\&cmdHelp,
);
my $helpMessage=<<HELP;
Who asked for help? Here's all the other special commands I know...
*jenkins - show status of our platform tests
*alert - add this room to get automatic notification of build status
*noalert - cancel notifcations
*help - displays this message
HELP
#status for jenkins tracking
my %builds;
my $lastJenkinsCheck=0;
my $alertRoomsFile='alert.rooms';
my $alertRooms={};
#store jenkins state
checkJenkins();
#because that was our first fetch, we'll have flagged everything as changed
#but it hasn't really, so we reset those flags
resetJenkinsChangeFlags();
#remember rooms we're supposed to alert
loadAlertRooms();
#attach to skype and enter message loop
my $skype = SkypeAPI->new();
my $attached=$skype->attach();
$skype->register_handler(\&onEvent);
$skype->register_idler(\&onIdle);
$skype->listen();
exit;
#here are the command handlers
sub cmdJenkins
{
my ($chatId, $args)=#_;
my $message="";
foreach my $build (keys(%builds))
{
$message.=formatBuildMessage($build)."\n";
#reset changed flag - we've just show the status
$builds{$build}->{'changed'}=0;
}
chatmessage($chatId, $message);
}
sub cmdAlert
{
my ($chatId, $args)=#_;
addChatroomToAlerts($chatId,1);
}
sub cmdNoAlert
{
my ($chatId, $args)=#_;
addChatroomToAlerts($chatId,0);
}
sub cmdHelp
{
my ($chatId, $args)=#_;
chatmessage($chatId, $helpMessage);
}
#simple helper to transmit a message to a specific chatroom
sub chatmessage
{
my ($chatId, $message)=#_;
my $commandstr="CHATMESSAGE $chatId $message";
my $command = $skype->create_command( { string => $commandstr} );
$skype->send_command($command);
}
#refreshes our copy of jenkins state, and will flag any builds
#which have changed state since the last check
sub checkJenkins{
my $xml = get($jenkinsRss);
my $tree = parse($xml);
my $items=$tree->{'channel'}->[0]->{'item'};
foreach my $item (#{$items})
{
my $title=$item->{'title'};
my $link=$item->{'link'};
my $built=$item->{'lastbuilddate'};
#print Dumper($item);
if ($title=~m/^(.*?) #(\d+)\s*(.*)$/)
{
my $build=$1;
my $buildnumber=$2;
my $status=$3;
#print "$build\n$buildnumber\n$status\n$link\n$built\n\n";
#build in progress, ignore
if (!exists($builds{$build}))
{
$builds{$build}={};
$builds{$build}->{'status'}='';
$builds{$build}->{'changed'}=0;
}
$builds{$build}->{'name'}=$build;
if ($status eq '(?)')
{
$builds{$build}->{'in_progress'}=1;
next; #don't update until complete
}
else
{
$builds{$build}->{'in_progress'}=0;
}
#is this status different to last status?
if ($builds{$build}->{'status'} ne $status)
{
$builds{$build}->{'changed'}=1;
}
$builds{$build}->{'status'}=$status;
$builds{$build}->{'build_number'}=$buildnumber;
$builds{$build}->{'link'}=$link;
$builds{$build}->{'built'}=$built;
}
}
#print Dumper(\%builds);
}
#generates a string suitable for displaying build status in skype
sub formatBuildMessage
{
my ($build)=#_;
my $status=$builds{$build}->{'status'};
my $smiley=":)";
if ($status=~m/broken/)
{
$smiley="(devil)";
}
elsif ($status=~m/\?/)
{
#this means the build is being retested, we should skip it
$smiley=":|";
}
my $message='';
if ($builds{$build}->{'in_progress'})
{
$message=":| $build - rebuild in progress..."
}
else
{
my ($y,$mon,$d,$h,$m,$s) = $builds{$build}->{'built'} =~ m/(\d\d\d\d)-(\d\d)-(\d\d)T(\d\d):(\d\d):(\d\d)Z/;
my $time = timegm($s,$m,$h,$d,$mon-1,$y);
my $age=time()-$time;
my $mins=round($age/60);
my $hrs=round($age/3600);
my $days=round($age/86400);
my $niceage;
if ($mins<=2)
{
$niceage="a few moments ago";
}
elsif ($mins<120)
{
$niceage="$mins minutes ago";
}
elsif ($hrs<48)
{
$niceage="$hrs hours ago";
}
else
{
$niceage="$days days ago";
}
$message="$smiley $build last built $niceage $status";
}
return $message;
}
#forget any changes we've flagged
sub resetJenkinsChangeFlags
{
foreach my $build (keys(%builds))
{
$builds{$build}->{'changed'}=0;
}
}
#checks for builds which have changed state. Can be called
#often, it will only kick in if 60 seconds have elapsed since
#last check
sub checkForJenkinsChanges
{
my $now=time();
if (($now-$lastJenkinsCheck) < 60)
{
#no need, we fetched it recently
return;
}
checkJenkins();
my $message='';
foreach my $build (keys(%builds))
{
if ($builds{$build}->{'changed'})
{
$builds{$build}->{'changed'}=0;
$message.=formatBuildMessage($build)."\n";
}
}
if (length($message))
{
foreach my $chatId (keys(%$alertRooms))
{
chatmessage($chatId, $message);
}
}
$lastJenkinsCheck=$now;
}
#adds or removes a room from the alerts
sub addChatroomToAlerts
{
my($chatId, $add)=#_;
if ($add)
{
if (exists($alertRooms->{$chatId}))
{
chatmessage($chatId, "/me says this room is already getting alerts");
}
else
{
$alertRooms->{$chatId}=1;
chatmessage($chatId, "/me added this chatroom to jenkins alerts");
}
}
else
{
delete($alertRooms->{$chatId});
chatmessage($chatId, "/me removed this chatroom from jenkins alerts");
}
store $alertRooms, $alertRoomsFile;
}
sub loadAlertRooms
{
if (-e $alertRoomsFile)
{
$alertRooms = retrieve( $alertRoomsFile);
}
}
# Skype event handler
sub onEvent {
my $skype = shift;
my $msg = shift;
#my $command = $skype->create_command( { string => "GET USERSTATUS"} );
#print $skype->send_command($command) , "\n";
#print "handler: $msg\n";
#an inbound chat message is either
#MESSAGE 13021257 STATUS RECEIVED (from others)
#MESSAGE 13021257 STATUS SENT (from us)
if ($msg =~ m/MESSAGE (\d+) STATUS (SEND|RECEIVED)/)
{
my $msgId=$1;
#get message body
my $commandstr="GET CHATMESSAGE $msgId BODY";
my $command = $skype->create_command( { string => $commandstr} );
my $output=$skype->send_command($command);
#if its a message for us...
if ($output =~ m/MESSAGE $msgId BODY \*([^\s]*)\s*(.*)/i)
{
my $botcmd=$1;
my $botargs=$2;
$commandstr="GET CHATMESSAGE $msgId CHATNAME";
$command = $skype->create_command( { string => $commandstr} );
$output=$skype->send_command($command);
if ($output =~ m/MESSAGE $msgId CHATNAME (.*)/)
{
my $chatId=$1;
if (exists($commands{$botcmd}))
{
$commands{$botcmd}->($chatId, $botargs);
}
else
{
chatmessage($chatId, "/me suggests trying *help as the robot didn't understand *$botcmd");
}
}
}
}
}
#skype idle handler
#Note - SkypeAPI.pm was modified to support this
sub onIdle {
my $skype = shift;
checkForJenkinsChanges();
sleep 0.1;
}
Step 3 - Run the bot
If you've saved this as robot.pl, just open a console window and perl robot.pl should get it running. Skype will ask you if perl.exe should be allowed to communicate with it, and once you confirm that, you're good to go!
Go into a team chatroom and type *jenkins for an summary of latest builds, and register the room for alerts of build changes with *alert
Perfect :)
Though the answer provided above is a working solution, I think that is a bit old and no tools were available at the time the question was asked. There is a plugin available for Jenkins to integrate with skype:https://wiki.jenkins-ci.org/display/JENKINS/Skype+Plugin
This plugin can be used to send notification to a particular user or a group chat. I've been using it for a while and it works great. I primarily use linux servers for my Jenkins servers and I have a dedicated windows server with skype installed and logged in as a bot user. The only configuration needed for running skype in a slave is that it requires a label called "skype". With this label Jenkins will detect the slave and route skype notification to that slave.
One can use Sevabot (https://sevabot-skype-bot.readthedocs.org/) project to setup a HTTP based interface for Skype and then use Jenkin's Notification Plugin:
Example: https://sevabot-skype-bot.readthedocs.org/en/latest/jenkins.html