I am building a project which requires constant connection with the server.
There are two major ways to achieve this:
Ajax pull
Ajax push
I have to decide between pinging a server (expensive) and maintaining keep-alive connections (firewalls block that.)
I was thinking about the live video streams. They are not keep-alive connections, nor frequent pings.
Is it possible, to send data, like JSON strings through rtmp?
It would be theoretically possible to implement RTMP's AMF3 and AMF0 Message types to carry the data. RTMP [Wikipedia]
The problem is that using a protocol typically used for streaming video might get your connection blocked or throttled by some service providers that limit such protocols to conserve bandwidth (and prevent employees from watching internet videos at work).
Maybe this article may be of some use to you. It explains how to set up an RTMP server with nginx.
From the article:
nginx is an extremely lightweight web server, but someone wrote a RTMP module for it, so it can host RTMP streams too. However, to add the RTMP module, we have to compile nginx from source rather than use the apt package. Don't worry, it's really easy. Just follow these instructions. :)
One comment on this article by a user named 'stefaniuk' linked to a github respitory for this that I think you should look in to. Check it out here.
Related
Basically, I'm building a Dropbox clone that will avoid cloud storage. Ok, I'm not building it, but trying to estimate the amount of work needed.
Been reading different p2p options here on SO, but actually, there are very little topics on centralised p2p connections and how to build them from ground-up. I'm not even sure if it's appropriate to call it a p2p at all.
From ActionScript background I know that it can establish an UDP connection between 2 different clients across the globe with provided centralised server (RTMFP). It's highly abstracted, it doesn't even require to open ports and clients don't know the IPs of each other. So the subset of given options is quite limited.
Anyway, I need create a server-side app and a client-side app that will try to sync files between connected clients. I've read that socket connections are used for file transfers. And the questions here are:
How to pair the clients?
What should server do?
What should client do?
Thank you.
NB
Establishing connections and file syncing solutions are out of the question.
I am working on a website that displays some data from DB that changes frequently (Status of a queue and a chat conversation). My current setup is Apache/PHP/MySQL. Naturally I would like to avoid polling the server every x seconds since this does not scale well. I would like to do reverse ajax long polling, however, I've read that Apache does not work well with this since it quickly runs out of worker threads. There are many other web servers out there that get around this problem: nginx, tornado, etc. However, my problem is, PHP is the ONLY server-side scripting language I know. Also I've already written some PHP scripts so I'd like to keep them if I can. I am ok with switching server so long as I can still use PHP.
But after doing some more research, I've read that people say PHP (PHP-FPM?) also creates a process for every request made, which means if I have hundreds/thousands of open connections, there will be hundreds/thousands of processes, which will be problem as well.
Can I conclude that there's no good scalable ways to make long polling websites using PHP? Should I abandon PHP and learn another server scripting language? I can continue developing long polling using my current setup (Apache/PHP) for now but I don't want the choice of scripting language to pose any limitation on the scalability of my system when I deploy. So what should I do? I am not very experienced with web programming, so if any gurus out there can give me some pointers I'd appreciate it! Thank you!
PHP runned in php-fpm mode will still have limitations, especially if your code is eating a lot of memory. You won't be able to run thousands of parallel processes without some available memory. But it usually perform faster than mod_php, and at least HTTP request that do not need PHP are handled by the webserver, and if that webserver is nginx you'll get a lot more HTTP requests available in parallel.
With php-fpm you will also have a queue of waiting requests, that may be usefull in case a temporary big traffic, as at least requests are queued, not rejected.
Now the long polling operations are OK with nginx (or others, that's an example), but not with PHP. PHP is not built to be a long-running server, each request is a new process, it's really not the right choice for a KeepAlive thing. But "Divide ut regnes" (divide and rule). Your long polling tasks could run near your PHP application, but without your PHP application.
As an example look at the jappix project, this is a PHP project. But you need to put somewhere an XMPP server (like ejabberd), and a BOSH server with nginx as a proxy on port 80 to that BOSH server (so you have the xmpp chat protocol on port 80, via nginx and ejabberd, and nothing on the PHP side for that). The problem is then to connect your application authentification, identification, and such, and this will have to be done by extending the XMPP server configuration (so that it use the same LDAP server as your PHP app for example).
Your second long polling problem is the status of a queue. You may find some XMPP extensions for that, maybe. Or you may perform regular ajax queries on the queue. One of the useful technique to avoid the big number of ajax requests on your PHP application is to reschedule the next ajax check on the ajax callback of the check, based on the Fibonacci numbers (it's an example). So the first time the next ajax call will be scheduled 1 minutes after, next time 2 minutes, then 3m, 5m, 8m, 13m, 21m, 34m, 55m, 89m, 144m, etc. The idea is that it's maybe important to check new messages incoming 1 minute after a page load. As the user is still reading the same page (or drinking a coffee, talking to a friend, going to holidays without switching off his computer, etc), we can delay more and more the next checks. Is a way of assuming the user is not really active. Note that you could detect user activity by other means and alter the rescheduling.
PHP is nor right for long polling, Comet and reverse ajax technologies. You should use Node.js
I want to communicate xml serialized objects from the server to the client and the other way arround. Now it is (probably) easy to invoke methods from a mobile client (compact framework) using WCF, but is there a way so that the server can invoke methods on the client side or some other way to pull data from the client? I know that callback contracts are not available in the compact framework as you can see here: http://blogs.msdn.com/andrewarnottms/archive/2007/09/13/calling-wcf-services-from-netcf-3-5-using-compact-wcf-and-netcfsvcutil-exe.aspx
Originally I thought of socket programming and of developing this by myself, then someone here mentioned WCF. But it seems like WCF only would work in a non mobile environment as I need callbacks.
Anyone can help me with this? Is it possible to develop a two way communication with a desktop server and multiple mobile clients using WCF, or will I have to do socket programming?
Thanks for any advice or any kind of help!
at ctacke
Thank you for your help. I actually stumbled across your Padran web server before.Havent really checked it out yet. But I definitely will do that later on. Anyway, a socket solution does not seem that bad at the moment. In the meanwhile I figured that it is quite easy to send data from multiple clients to a 'socket server'. If I can manage those connections somehow I can send data back to the clients. And then I would have to come up with some kind if protocol which handles the data or commands I send over the network... I guess the hardest part would be to make up such a protocol as I do not have a clue about that atm...
Even if you go to sockets it might be a bit difficult due to routing, carrier filtering and NAT translations (you've not mentioned what your actual network topology is). This is the reason that most mobile applications have to poll the server, even if it is a "push" paradigm (like Exchange's push mechanism, where the client actually polls).
Generally speaking, unless you're on something like a local wireless network where you have solid, routable, unfiltered network access, the client should periodically call the server and ask if the server has data. If it does, then it pulls the data from the server.
EDIT
Now that we know a little more about your topolgy from your comment, I can steer you a little more. Unfortunately Microsoft has not made it easy for Windwos CE devices to host services (WCF or otherwise). There is, in theory, the required infrstructure to build up your own WCF channel and actually host a service, but it's not a trivial task. I looked into it quite some time ago and figured it was a couple months of work and that would have been with the assistance of someone in Redmond that knew how the existing Exchange channel works.
Personally I'd opt for hosting a REST-based web service using our Padarn web server because it's simple to do and I've done it for quite a number of clients now. I realize that it's a little self-serving to propose Padarn as a solution but the entire reason I implemented custom IHttpHandlers in Padarn was because I couldn't find anything else out there that really provided any easy way for a CE device to host it's own services and it's a problem we often have to provide a solution for.
The other options would be things like a proprietary socket solution, hosting an FTP server on the device, using the (abhorrent, IMO) MS-provided HTTP server with ISAPI, Telnet or something along those lines. All of them seem either a hack, a lot of work or both.
I am working on a private messasing functionality for my site,
is there a way to get push notifications from the server, that the user has received a new message?
the only technique I know is to constantly poll the server via ajax to see if there are new messages, and reload the messenger window if needed.
thanks for any feedback!
UPD: so far the following directions have been identified:
1) Comet
2) BOSH - Bidirectional-streams Over Synchronous HTTP
3) XMPP - this is what google talk is based on
You might want to look into XMPP and BOSH. Very comet like and ejabberd combined with nginx can maintain 1000's of connections on a very small box.
Take a look at one of my projects www.vooices.us to see what we have done with XMPP and BOSH.
http://www.ejabberd.im/
http://github.com/ssoper/jquery-bosh/tree/master JQueryBosh - to enable quick and easy access to XMPP over HTTP
It is relativly easy to set up and install.
Paul.
There isn't any way for the server to push information to the client, but you could always use Comet. Yes, you'd be polling the server, but not constantly. You'd keep each connection open for about 30 seconds (that's the interval I've always seen used) and then open a new connection when needed.
EDIT: Adobe Flash allows persistent connections to the server, so you could use that. See this article for details.
Sounds like the Comet probably links to some useful stuff (Look into Google's GTalk):
http://alex.dojotoolkit.org/2006/02/what-else-is-burried-down-in-the-depths-of-googles-amazing-javascript/
There are a lot of good libraries that help you to do this
XMPP - BOSH is a good combination which I use on my websites
XMPP servers
Ejabberd, Openfire I use ejabberd
You can use Strophe along with Ejabberd to capture notifications in javascript. If you want to send notifications from code
There are libraries like JAXL and XMPPHP that let you do this, but this is in PHP.
How is Apache in respect to handling the c10k problem under normal conditions ?
Say while running very small scripts with little data, or do I need to scale out if I use Apache?
In the background heavy lifting is done by a few servers running specialized software that processes the requests but I'd like to use Apache as a front. Is this a viable plan?
I consider Apache to be more of an origin server - running something like mod_php or mod_perl to generate the content and being smart about routing to the appropriate system.
If you are getting thousands of concurrent hits to the front of your site, with a mix of types of data (static and dynamic) being returned, you may find it useful to put a more optimised system in front of it though.
The classic post-optimisation problem with Apache isn't generating the dynamic content (or at least, that can be optimised for early in the process), but simply waiting for a slow client to be able to receive the bytes that are being sent. It can therefore be a significant advantage to put a reverse proxy, in the form of Squid or Nginx, in front of the servers to take over the 'spoon-feeding' of the slow network clients, while allowing the content production to happen at full speed, and at local network speeds - 100Mb/sec or even gigabit speeds - if it even has to traverse a network at all.
I'm assuming you've probably seen this data, but if not, it might give you some idea.
Guys, imagine that you are running web server with 10K connections (simultaneous). How could it be?
You've got many many connections per second
Dynamic content
Are you sure that your CPU can handle that many PHP sessions for example? I guess no, so why are you thinking about C10K problem? :D
Static content - small files
And still soo many connections? On single server? Probably you've got problems with networking/throughput too or you are future competitor of Google. Use lighttpd which addresses C10K problem and is stable - fly light. Using Apache for only static files for large sites is obvious.
Your clients are downloading large files for a large time - static content
ISO images, archives etc
If you are doing it via web server - FTP may be more appropriate.
Video streaming
Use lighttpd or specialized software. And still... What about other resources?
I am using Linux Virtual Server as load balancer in front of apache servers (with specific patches for LVS-NAT) and I am happy :) This string is an answer you want to hear.