so I am currently working in heroku and I need the task to programmatically restart if certain errors arise, so I used the heroku restart API to do so. The problem is that my program is restarting every 10 seconds or so, and it seems that heroku is letting the other web tasks stay online which leads to them later on crashing. I was wondering if there was anyway to only run one task at a time and skip their 60/30 seconds waiting period for when you restart a task. I read somewhere about One - off tasks, but I wasn't sure. I am sorry for the formatting of my question, it is my first post on StackOverflow. Thank you.
Related
I have a project with obout 30 spiders, all scheduled via cron job. Whenever I want to deploy a project I git push to production where a hook will put the files in place.
Now I came accross scrapyd which seems to do both in a more soffisticated way by egifying the scraper and deploying it to the production environment. Looking at the code it seems that this project has come to a halt about 3 years ago. I am wondering if there is an advantage to switch to scrapyd and what the reason is for this code to be so old and no longer under development. Scrapy itself receives regula updates in contrast.
Would you advice to use scrapyd and if yes, why?
I've been using scrapyd for about 2 years, and I do prefer to use it over just launching your jobs using scrapy crawl:
You can set the number of scrapers that can run at the same time using `max_proc_per_cpu. Any scrapers you launch when the max is reached, are put in a queue and launched when a spot is available.
You have a minimalistic GUI in which you can check the queues & read the logs.
Scheduling spiders is easily done with the api-calls. Same for listing the spiders, cancelling spiders, ...
You can use http cache even when running multiple spiders at the same time
You can deploy on multiple servers at once if you want to spread out your crawls over different servers
I have a Rails app running on Apache/Passenger. It has a rufus-scheduler cron job that runs in the background and sends out notifications via email.
When I am running the app in development on the WEBrick server, the emails are sent like they are supposed to be sent.
When I start up the app in production on Apache/Passenger, the emails don't get sent. In the production logs it doesn't show any logs for rufus-scheduler.
I'm stuck in this problem. Your help will be appreciated, thank you in advance.
The easiest solution is to set PassengerSpawnMethod to direct. The Phusion Passenger documentation explains why this solves it: http://www.modrails.com/documentation/Users%20guide%20Apache.html#spawning_methods_explained
In particular, take a look at section "15.4. Smart spawning gotcha #2: the need to revive threads".
Whew...ok, been wrestling with this for a while and I can't figure out what is going on.
I am new to Azure caching, but at this point I have read a good bit and I think I have it setup right, but something is obviously wrong so what do I know?
Ok, so first I setup a dedicated caching web worker role using this fine tutorial: http://berniecook.wordpress.com/2013/01/27/distributed-caching-in-azure-cache-worker-role/
I have an ASP.net MVC 4 website that is supposed to be using it.
I have my solution set to multiple starting projects with my cloud caching project set to start first, but no matter what I do, I get the "role discovery data is unavailable".
Sometimes in my output log I get that the Role Environment failed to initialize, but not very often. Most of the time the output log says that is succeeds. Regardless of that, I still get the error above.
I was thinking that maybe the issue was because I was running on local azure storage and compute emulators, so I reconfigured and published the Cloud Service to Azure to see if that helped.
It didn't...
The fun part is that there have been exactly 2 times when it suddenly worked (both when I was working locally). 2 times about of about 100. I didn't do anything different...just ran the debugger and poof, it all worked. This at least lends a bit of credit that it is actually setup correctly.
Needless to say, this is putting a huge damper on my productivity so any advice would be appreciated.
Update
Ok, I have figured out a workaround of sorts...I have learned that the reason that it consistently failed was because the development web server was holding onto a file which prevented the caching server to launch correctly.
The workaround is to stop the web server each and every time I want to recompile and run the code. This is obviously not ideal, so any ways to make this more reliable would be appreciated.
Thanks,
David
I don't know if this helps but I find that if I don't shut down the both the storage and compute emulator, I get weird errors, so after doing an F5 and closing the browser down, I manually shut down both emulators
My problem is, that I am running a webapp on Tomcat7 on a 2GB 2Core VPS and when the app crashes with heapoverflow exception Tomcat doesn't start or stop. When I try to access it via the browser it just hangs.
If I try to stop it looks like it stops but if I do it repeatedly it is supposed to throw an exception because it's already stopped, but this is not the case. Only after 15 minutes am I able to start it up again.
I know that I should solve the heapoverflow problem, but the thing is I am simply not able as the framework I am using turned out to be nod thread safe and I have to create a new instance of a memory heavy object for every request :( I tried to come round the problem by adding some basic controller which only allows certain amount of objects to be created and the rest of the request are redirected to a Sorry-page. Since it is for a UNI project and the main goal is not being able to serve hundreds of users it's not important to make the framework thread safe.
Still I need to know why Tomcat does it after it crashes. Also it would help if you would have a better idea how to limit the number of these objects being in the memory at the same time.
++++++++++
At the moment the only solution I can see is to get say 2GB of more RAM so I will be able to serve some 15-20 users at a time.
Generally after tomcat crashes with out of memory error it is in unusable state and the process has to be killed either manually or by using
catalina.sh stop -force
For a discussion on how to prevent those out of memory errors see
Dealing with "java.lang.OutOfMemoryError: PermGen space" error
Though ultimately you may need to throw more hardware at the problem.
This might also come handy:
Can I auto restart tomcat jvm on out of memory exception
I am running Unicorn on Heroku. I notice when we scale web dynos up. These new web dynos are being accessed right after it is spawned. In the logs we are getting:
Request timeout" error with 30 seconds limit (i.e. service=30000ms)
As soon as a dyno starts, traffic will be sent to it. If Unicorn is still spinning up child processes as these requests arrive, then they will likely time out, especially if your app takes a while to boot.
Is there a way to spawn Unicorn child processes without them being accessed by requests, until the app is fully loaded?
This is actually something else and something I've encountered before.
Essentially, when you scale you're seeing your dyno get created and your slug being deployed onto it. At this point your slug is initialised. The routing mesh will start sending requests through the moment this has happened, as it sees the dyno as up and ready to rock.
However, (and this is where I found the problem), it takes time for your application to spin up to respond to the request (Unicorn is up, Rails is still initialising) and you get the 30 second timeout.
The only way I could fix this was to get my app starting up in less than 30 seconds repeatedly which I finally achieved by updating to a current version of Rails. I've also found some increases by also updating to try running on Ruby 1.9.3