I'm looking to automate how my bots are updated. They're hosted on a VPS from GalaxyGate and kept alive using PM2. I use VSCode to develop them, then I push to GitHub and pull the code from there to the VPS to update them directly in production. However, this currently has to be done manually and I always forget the git commands that I'm supposed to run.
I'm looking to have this process automated; ideally the server would periodically run some sort of script to update all bots that are hosted on it. It would fetch GitHub and apply the new changes (if any) and properly restart the bots, so I have a few questions:
Is this possible, and if so, how would I go about doing that?
Are there any risks/downsides with such a system?
Any help would be appreciated
Related
I have a project with obout 30 spiders, all scheduled via cron job. Whenever I want to deploy a project I git push to production where a hook will put the files in place.
Now I came accross scrapyd which seems to do both in a more soffisticated way by egifying the scraper and deploying it to the production environment. Looking at the code it seems that this project has come to a halt about 3 years ago. I am wondering if there is an advantage to switch to scrapyd and what the reason is for this code to be so old and no longer under development. Scrapy itself receives regula updates in contrast.
Would you advice to use scrapyd and if yes, why?
I've been using scrapyd for about 2 years, and I do prefer to use it over just launching your jobs using scrapy crawl:
You can set the number of scrapers that can run at the same time using `max_proc_per_cpu. Any scrapers you launch when the max is reached, are put in a queue and launched when a spot is available.
You have a minimalistic GUI in which you can check the queues & read the logs.
Scheduling spiders is easily done with the api-calls. Same for listing the spiders, cancelling spiders, ...
You can use http cache even when running multiple spiders at the same time
You can deploy on multiple servers at once if you want to spread out your crawls over different servers
I am trying to host an angular server side rendering app. Whenever I try this command npm run serve:ssr it runs fine and after a while it stops automatically. I just have 2 questions.
How do i keep it running forever ?
Is that the right way to do for a production ready app?
Please suggest.
I would follow angular's advice and discourage node and express as servers. But sometimes you have to do it. If you do care about security ( a lot ) I would crawl the entire site and drop the results in a static NGINX server.
In either case you should use angular universal with a grain of salt.
And to keep things runing forever you can turn the npm serve:ssr into a deamon (unix) so that the system keeps it on. Or simple change it to : while true; do npm serve:ssr; done
I Have a VPS that is scheduled to restart Several times.
On the start-up, i need to the browser to automatically run some simple tasks, like load a php script, clear history, change some simple random parameters.
I tried to accomplished this with auto clicker and macro creator programs like perfect automation , mouse recorder and a few others, and while it works fine, when I'm connected to the VPS, it does not work at all when i restart the machine, which is the objective of this problem: to full automate initial browser tasks on the machine start-up, without login.
Is there any fullproof software for this kind of stuff ? the time I'm wasting to try to accomplish this is becoming ridiculous
I'll be using RTC in the near future here at work. My question is: where does it put the files the team members will be working on? I understand that each programmer will work on the projects files and they will push the changes to the main repository. We have a local web server where we test our work (php). So, do we have to configure RTC to publish the files to the web server? or the RTC server must be installed in the webserver so it can save the files?
We use Rational Team Concert almost exactly as you describe, and it works brilliantly. My small team of web developers collaborates on website source code and delivers it to two different streams depending on its readiness: production-stream and staging-stream. Then we have defined two builds that check out the source code, move some things around, and push the files to the web servers via SCP. So, with a few clicks we kick off a staging build, watch it finish in about two minutes and everyone can see the changes on the staging server. When the code is ready for prime-time, the change sets are delivered to production-stream and the production build is kicked off, which is configured to copy the files to the production web server.
But even before a staging or production build is run, any of us can simply configure a local web server in RTC using the Eclipse PDE and Web Tools add-ons and see the site running in localhost as we develop.
All our work is done within Rational Team Concert, from planning, to bug tracking, to source control, to builds. It's very well-suited for website management.
Your understanding is correct - you work on files locally, and they get uploaded on to the server when you checkin. Bear in mind that checkin in RTC terms really means back-up your files to the server, it is a Deliver command that shares the files with others (it is worth a quick look at the articles on jazz.net that explains how SCM works).
One way to pubish to your php server is to make that part of a build, or a build in its own right (which RTC also handles - in conjunction with your favourite build tool). The build would copy the files to the php server. The advantage of doing this as a build is you will know exactly what versions of your files are being copied, and you will be able to reproduce this copy at any point in the future.
You do not need to install the RTC server on the php server.
You can also try posting on the forums on http://jazz.net/ if you have questions on RTC.
Hope that helps.
Another alternative would be to use the command line interface to accept all changes into a workspace and run that with a cron job.
To handle discarded change sets, you'd probably want to use something like:
scm workspace replace-components <workspace-name> stream <uuid-of-stream> --all
after you had initially loaded the workspace on your web server.
I need a web server that allows me to remove a file after it has been successfully downloaded once. Is there any way to do this with apache?
Is there another web server I can use for this task? I had already looked into Tornado for this purpose, but couldn't find a way to get an event to fire as soon as the download finished. the on_connection_close would only fire when I shut down the server.
I'd prefer something PHP or Python-based if I have to code it myself.
you might be able to accomplish this with LUA (lighttpd + mod_magnet) - but im not sure... besides that im not aware of any modules that has event triggers on "downloads" there are some that work with uploads like the mod_uploadprogress in lighttpd. im sure nginx has some similar modules... besides that... i think the only solution here is:
http://redmine.lighttpd.net/wiki/lighttpd/HowToWriteALighttpdPlugin