Just want to know if someone already get some issues with changeFeedWorker after that cosmos db reach 50Gb size limit for one partition and automatically split it into 2 partitions.
Since this split we have observed that after adding many new items to the db, changefeedworkers was not always triggered and our view that should get changes is partially updated.
In the lease container we can see now that we moved to 2 "LeaseToken" (1 and 2) was 0 before.
If someone has an idea where to look as it was working fine before the db split.
Here is how I start my worker:
async Task IChangeFeedWorker.StartAsync()
{
await _semaphoreSlim.WaitAsync();
string operationName = $"{nameof(CosmosChangeFeedWorker)}.{nameof(IChangeFeedWorker.StartAsync)}";
using var _ = BeginChangeFeedWorkerScope(_logger, operationName, _processorName, _instanceName);
try
{
if (Active)
{
return;
}
Container eventContainer = _cosmosClient.GetContainer(_databaseId, _eventContainerId);
Container leaseContainer = _cosmosClient.GetContainer(_databaseId, _leaseContainerId);
_changeFeedProcessor = eventContainer
.GetChangeFeedProcessorBuilder<CosmosEventChange>(_processorName, HandleChangesAsync)
.WithInstanceName(_instanceName)
.WithLeaseContainer(leaseContainer)
.WithStartTime(_startTimeOfTrackingChanges)
.Build();
await _changeFeedProcessor.StartAsync();
Active = true;
_logger.LogInformation(
"Change feed processor instance has been started.",
_processorName, _instanceName);
}
catch (Exception e)
{
_logger.LogError(e,
"Starting of change feed processor instance has failed.",
_processorName, _instanceName);
_changeFeedProcessor = null;
throw;
}
finally
{
_semaphoreSlim.Release();
}
}
I've a Redis cluster with 3 nodes; 1 is the master and the other 2 are slaves, holding the replica of the master. When I kill the master instance, Redis Sentinel promotes another node to be the master, which starts to accept writes.
During my tests I noticed that once the new master is promoted, the first operation in Redis with SE.Redis fails with:
StackExchange.Redis.RedisConnectionException: SocketFailure on GET
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote
host. ---> System.Net.Sockets.SocketException: An existing connection
was forcibly closed by the remote host
To avoid it, I've implemented a retry logic as below. Is there any better alternative?
private RedisValue RedisGet(string key)
{
return RedisOperation(() =>
{
RedisKey redisKey = key;
RedisValue redisValue = connection.StringGet(redisKey);
return (redisValue);
});
}
private T RedisOperation<T>(Func<T> act)
{
int timeToSleepBeforeRetryInMiliseconds = 20;
DateTime startTime = DateTime.Now;
while (true)
{
try
{
return act();
}
catch (Exception e)
{
Debug.WriteLine("Failed to perform REDIS OP");
TimeSpan passedTime = DateTime.Now - startTime;
if (this.retryTimeout < passedTime)
{
Debug.WriteLine("ABORTING re-try to REDIS OP");
throw;
}
else
{
int remainingTimeout = (int)(this.retryTimeout.TotalMilliseconds - passedTime.TotalMilliseconds);
// if remaining time is less than 1 sec than wait only for that much time and than give a last try
if (remainingTimeout < timeToSleepBeforeRetryInMiliseconds)
{
timeToSleepBeforeRetryInMiliseconds = remainingTimeout;
}
}
Debug.WriteLine("Sleeping " + timeToSleepBeforeRetryInMiliseconds + " before next try");
System.Threading.Thread.Sleep(timeToSleepBeforeRetryInMiliseconds);
}
}
}
TLDR: don't use Sentinel with Stackexchange.Redis as Sentinel support is still not implemented in this client library.
See https://github.com/StackExchange/StackExchange.Redis/labels/sentinel for all the open issues, there is also a pretty good PR open for ~1 year now.
That being said, I also had relatively good experience with retries, but I would never use that approach in production as it is not reliable at all.
I am trying to get gmail attachments using batch requests on the gmail API. It sometimes works, but if I rerun it, I'll get errors a large percentage of the time. I'd like to catch the errors and take appropriate action - like exponential backoff, which will (hopefully) fix the problem. But it's not working.
Here is the code:
for ($n = 0; $n < 5; ++$n) {
try {
$results = $batch->execute();
break;
} catch (Google_Service_Exception $e) {
if ($e->getError() != '') { //placeholder
// Apply exponential backoff.
usleep((1 << $n) * 1000 + rand(0, 1000));
}
}
}
for ($i=1; $i < $stop+1; $i++) {
$message = $results['response-'.$i];
if ($message2->getPayload()) { ...
The error logs show that sometimes I'm immediately trying to call Google_Service_Exception::getPayload. It's as if each response in the batch was itself a Google_Service_Exception object, or at least the first one.
How do I catch errors in a batch?
it is very first time i'm messing with sockets , and read many quotes that this is not for newbies.
so problem is i'm using php smpp library for sending SMS which works fine but after delivering two to three SMS delivery fails with following warning
Warning: stream_socket_sendto() [function.stream-socket-sendto]: An existing connection was forcibly closed by the remote host', and to make it work again i need to restart` apache.
Following is write function which throwing exception
public function write($buf) {
$null = null;
$write = array($this->handle_);
// keep writing until all the data has been written
while (strlen($buf) > 0) {
// wait for stream to become available for writing
$writable = #stream_select($null, $write, $null, $this->sendTimeoutSec_, $this->sendTimeoutUsec_);
if ($writable > 0) {
// write buffer to stream
$written = stream_socket_sendto($this->handle_, $buf);
if ($written === -1 || $written === false) {
throw new TTransportException('TSocket: Could not write '.$written.' bytes '.
$this->host_.':'.$this->port_);
}
// determine how much of the buffer is left to write
$buf = substr($buf, $written);
} else if ($writable === 0) {
throw new TTransportException('TSocket: timed out writing '.strlen($buf).' bytes from '.
$this->host_.':'.$this->port_);
} else {
throw new TTransportException('TSocket: Could not write '.strlen($buf).' bytes '.
$this->host_.':'.$this->port_);
}
}
}
Please anyone can put some light
It was the bug which i won't able to identify/ rectify. then i used an other library from http://www.codeforge.com/read/171085/smpp.php__html and it really saved me.
Our company uses Skype for communications, and I'd like to be able to send alerts to Skype chatrooms when a Jenkins build fails (and when it recovers too).
How can I do this?
I've done this using the Skype Public API
What I did was write a Perl script which uses the SkypeAPI CPAN module to handle the communications with Skype. It's a little clunky, as the script needs to run on a desktop which is running Skype. I run it on my own desktop which is always on anyway, but this does mean the bot appears to be 'me' to the rest of our team.
The end result is fantastic - whenever a jenkins build changes state, the bot sends a message to any Skype chats which have registered an interest by typing *alert. Additionally, any developer can see and share the latest build status by typing *jenkins
Step 1 - Extending the SkypeAPI module
Now, the SkypeAPI module is pretty basic. It has a message loop in the listen() method which checks for new events from the Skype client, and sleeps for a moment if there none.
I wanted my script to hook into this loop so that I could have my bot periodically check Jenkins RSS feed, so I made the following modifications to SkypeAPI.pm after I had installed it with the ActiveState package manager:
I declared new property 'idler' along with the existing properties...
__PACKAGE__->mk_accessors(
qw/api handler_list stop_listen idler/
);
I added a method to set an 'idler' callback which the module will call instead of sleeping
sub register_idler {
my $self = shift;
my $ref_sub = shift;
$self->idler($ref_sub);
}
Finally I modified the message loop to call the idler if set
sub listen {
my $self = shift;
my $idler=$self->idler();
$self->stop_listen(0);
while (!$self->stop_listen) {
my $message;
{
lock #message_list;
$message = shift #message_list;
}
if (not defined $message) {
if ($idler)
{
$self->idler->($self);
}
else
{
sleep 0.1;
}
next;
}
for my $id (sort keys %{$self->handler_list}) {
$self->handler_list->{$id}->($self, $message);
}
}
}
Step 2 - write a robot script
Now the module is a little more capable, it's just a matter of writing a script to act as a bot. Here's mine - I've made a few edits from my original as it contained other irrelevant functionality, but it should give you a starting point.
All of the dependant modules can be installed with the ActiveState package manager.
use strict;
use SkypeAPI;
use LWP::Simple;
use Data::Dumper;
use dirtyRSS;
use Time::Local 'timegm';
use Math::Round;
use Storable;
#CHANGE THIS - where to get jenkins status from
my $jenkinsRss='http://username:password#jenkins.example.com/rssLatest';
my %commands=(
'jenkins' =>\&cmdJenkins,
'alert' =>\&cmdAlert,
'noalert' =>\&cmdNoAlert,
'help' =>\&cmdHelp,
);
my $helpMessage=<<HELP;
Who asked for help? Here's all the other special commands I know...
*jenkins - show status of our platform tests
*alert - add this room to get automatic notification of build status
*noalert - cancel notifcations
*help - displays this message
HELP
#status for jenkins tracking
my %builds;
my $lastJenkinsCheck=0;
my $alertRoomsFile='alert.rooms';
my $alertRooms={};
#store jenkins state
checkJenkins();
#because that was our first fetch, we'll have flagged everything as changed
#but it hasn't really, so we reset those flags
resetJenkinsChangeFlags();
#remember rooms we're supposed to alert
loadAlertRooms();
#attach to skype and enter message loop
my $skype = SkypeAPI->new();
my $attached=$skype->attach();
$skype->register_handler(\&onEvent);
$skype->register_idler(\&onIdle);
$skype->listen();
exit;
#here are the command handlers
sub cmdJenkins
{
my ($chatId, $args)=#_;
my $message="";
foreach my $build (keys(%builds))
{
$message.=formatBuildMessage($build)."\n";
#reset changed flag - we've just show the status
$builds{$build}->{'changed'}=0;
}
chatmessage($chatId, $message);
}
sub cmdAlert
{
my ($chatId, $args)=#_;
addChatroomToAlerts($chatId,1);
}
sub cmdNoAlert
{
my ($chatId, $args)=#_;
addChatroomToAlerts($chatId,0);
}
sub cmdHelp
{
my ($chatId, $args)=#_;
chatmessage($chatId, $helpMessage);
}
#simple helper to transmit a message to a specific chatroom
sub chatmessage
{
my ($chatId, $message)=#_;
my $commandstr="CHATMESSAGE $chatId $message";
my $command = $skype->create_command( { string => $commandstr} );
$skype->send_command($command);
}
#refreshes our copy of jenkins state, and will flag any builds
#which have changed state since the last check
sub checkJenkins{
my $xml = get($jenkinsRss);
my $tree = parse($xml);
my $items=$tree->{'channel'}->[0]->{'item'};
foreach my $item (#{$items})
{
my $title=$item->{'title'};
my $link=$item->{'link'};
my $built=$item->{'lastbuilddate'};
#print Dumper($item);
if ($title=~m/^(.*?) #(\d+)\s*(.*)$/)
{
my $build=$1;
my $buildnumber=$2;
my $status=$3;
#print "$build\n$buildnumber\n$status\n$link\n$built\n\n";
#build in progress, ignore
if (!exists($builds{$build}))
{
$builds{$build}={};
$builds{$build}->{'status'}='';
$builds{$build}->{'changed'}=0;
}
$builds{$build}->{'name'}=$build;
if ($status eq '(?)')
{
$builds{$build}->{'in_progress'}=1;
next; #don't update until complete
}
else
{
$builds{$build}->{'in_progress'}=0;
}
#is this status different to last status?
if ($builds{$build}->{'status'} ne $status)
{
$builds{$build}->{'changed'}=1;
}
$builds{$build}->{'status'}=$status;
$builds{$build}->{'build_number'}=$buildnumber;
$builds{$build}->{'link'}=$link;
$builds{$build}->{'built'}=$built;
}
}
#print Dumper(\%builds);
}
#generates a string suitable for displaying build status in skype
sub formatBuildMessage
{
my ($build)=#_;
my $status=$builds{$build}->{'status'};
my $smiley=":)";
if ($status=~m/broken/)
{
$smiley="(devil)";
}
elsif ($status=~m/\?/)
{
#this means the build is being retested, we should skip it
$smiley=":|";
}
my $message='';
if ($builds{$build}->{'in_progress'})
{
$message=":| $build - rebuild in progress..."
}
else
{
my ($y,$mon,$d,$h,$m,$s) = $builds{$build}->{'built'} =~ m/(\d\d\d\d)-(\d\d)-(\d\d)T(\d\d):(\d\d):(\d\d)Z/;
my $time = timegm($s,$m,$h,$d,$mon-1,$y);
my $age=time()-$time;
my $mins=round($age/60);
my $hrs=round($age/3600);
my $days=round($age/86400);
my $niceage;
if ($mins<=2)
{
$niceage="a few moments ago";
}
elsif ($mins<120)
{
$niceage="$mins minutes ago";
}
elsif ($hrs<48)
{
$niceage="$hrs hours ago";
}
else
{
$niceage="$days days ago";
}
$message="$smiley $build last built $niceage $status";
}
return $message;
}
#forget any changes we've flagged
sub resetJenkinsChangeFlags
{
foreach my $build (keys(%builds))
{
$builds{$build}->{'changed'}=0;
}
}
#checks for builds which have changed state. Can be called
#often, it will only kick in if 60 seconds have elapsed since
#last check
sub checkForJenkinsChanges
{
my $now=time();
if (($now-$lastJenkinsCheck) < 60)
{
#no need, we fetched it recently
return;
}
checkJenkins();
my $message='';
foreach my $build (keys(%builds))
{
if ($builds{$build}->{'changed'})
{
$builds{$build}->{'changed'}=0;
$message.=formatBuildMessage($build)."\n";
}
}
if (length($message))
{
foreach my $chatId (keys(%$alertRooms))
{
chatmessage($chatId, $message);
}
}
$lastJenkinsCheck=$now;
}
#adds or removes a room from the alerts
sub addChatroomToAlerts
{
my($chatId, $add)=#_;
if ($add)
{
if (exists($alertRooms->{$chatId}))
{
chatmessage($chatId, "/me says this room is already getting alerts");
}
else
{
$alertRooms->{$chatId}=1;
chatmessage($chatId, "/me added this chatroom to jenkins alerts");
}
}
else
{
delete($alertRooms->{$chatId});
chatmessage($chatId, "/me removed this chatroom from jenkins alerts");
}
store $alertRooms, $alertRoomsFile;
}
sub loadAlertRooms
{
if (-e $alertRoomsFile)
{
$alertRooms = retrieve( $alertRoomsFile);
}
}
# Skype event handler
sub onEvent {
my $skype = shift;
my $msg = shift;
#my $command = $skype->create_command( { string => "GET USERSTATUS"} );
#print $skype->send_command($command) , "\n";
#print "handler: $msg\n";
#an inbound chat message is either
#MESSAGE 13021257 STATUS RECEIVED (from others)
#MESSAGE 13021257 STATUS SENT (from us)
if ($msg =~ m/MESSAGE (\d+) STATUS (SEND|RECEIVED)/)
{
my $msgId=$1;
#get message body
my $commandstr="GET CHATMESSAGE $msgId BODY";
my $command = $skype->create_command( { string => $commandstr} );
my $output=$skype->send_command($command);
#if its a message for us...
if ($output =~ m/MESSAGE $msgId BODY \*([^\s]*)\s*(.*)/i)
{
my $botcmd=$1;
my $botargs=$2;
$commandstr="GET CHATMESSAGE $msgId CHATNAME";
$command = $skype->create_command( { string => $commandstr} );
$output=$skype->send_command($command);
if ($output =~ m/MESSAGE $msgId CHATNAME (.*)/)
{
my $chatId=$1;
if (exists($commands{$botcmd}))
{
$commands{$botcmd}->($chatId, $botargs);
}
else
{
chatmessage($chatId, "/me suggests trying *help as the robot didn't understand *$botcmd");
}
}
}
}
}
#skype idle handler
#Note - SkypeAPI.pm was modified to support this
sub onIdle {
my $skype = shift;
checkForJenkinsChanges();
sleep 0.1;
}
Step 3 - Run the bot
If you've saved this as robot.pl, just open a console window and perl robot.pl should get it running. Skype will ask you if perl.exe should be allowed to communicate with it, and once you confirm that, you're good to go!
Go into a team chatroom and type *jenkins for an summary of latest builds, and register the room for alerts of build changes with *alert
Perfect :)
Though the answer provided above is a working solution, I think that is a bit old and no tools were available at the time the question was asked. There is a plugin available for Jenkins to integrate with skype:https://wiki.jenkins-ci.org/display/JENKINS/Skype+Plugin
This plugin can be used to send notification to a particular user or a group chat. I've been using it for a while and it works great. I primarily use linux servers for my Jenkins servers and I have a dedicated windows server with skype installed and logged in as a bot user. The only configuration needed for running skype in a slave is that it requires a label called "skype". With this label Jenkins will detect the slave and route skype notification to that slave.
One can use Sevabot (https://sevabot-skype-bot.readthedocs.org/) project to setup a HTTP based interface for Skype and then use Jenkin's Notification Plugin:
Example: https://sevabot-skype-bot.readthedocs.org/en/latest/jenkins.html