Right now, I am hosting a site on a dedicated server, 8 GB ram, Intel Xeon E3 1230 V3. I am using long polling techniques in order to display information which gets added into a database consistently.
The problem is: so far, after around let's say 20 users come onto the site, it starts lagging and slowing down dramatically. I'm pretty sure the server is strong enough to handle way more people than that. Thus, I am not sure what exactly is the problem. Can long polling using Apache handle that many users? If not, how should I implement real-time information being displayed. And if it can, how should I configure Apache or anything in order to handle around 500-1000 concurrent users.
Any help is appreciated.
Related
We are currently testing an upgrade from CF11 to CF2018 for my company's intranet. To give you an idea how long this site has been running, our first version of CF was 3.1! It is still using application.cfm, and there is code from 1998, when I started writing this thing. Yes, 21 years -- I'm astonished, too. It is a hodgepodge of all kinds of older frameworks, too, including Fusebox.
Anyway, we're running Win 2012 VM connected to a SQL 2016 farm. Everything looked OK initially, but in the Week I've been testing, the server has come to a slowdown once (a page took more than 5 seconds to run, something that usually takes 100ms, no DB involvement), and another time, the server came to a grinding halt. The only way I could restart CF App service was by connecting to the server with another server via Services, because doing it via Remote Desktop was so slow.
Now keep in mind -- it's just me testing. This is a site that doesn't have a ton of users, but still, having 5 concurrent connections is normal and there are upwards of 200-400 users hitting this thing every day.
I have FusionReactor running on this thing now, so the next time a lockup happens, I will be able to take a closer look, but what do you think is the best way I can test this? Our site is mostly transactional, users going and filling out forms to put internal orders through. We also connect to XML web services and REST services; we also provide REST services, too. Obviously there's no way to completely replicate a production server's requests onto a test server, but I need to do more thorough testing. Any advice would be hugely appreciated.
I realize your focus for now is trying to recreate the problem on test. That may not be as easy as hoped. Instead, you should be able to understand and resolve it in production. FusionReactor can help, but the answer may well be in the cf logs.
You don't mention assessing the logs at the time of the hangup. See especially the coldfusion-error log, for outofmemory conditions.
You mention raising the heap, but the problem may be with the metaspace instead. If so, consider simply removing the maxmetaspace setting in the jvm args. That may be the sole and likely cause of such new and unexpected outages.
Or if it's not, and there's nothing in the logs at the time, THEN do consider FR. Does IT show anything happening at the time?
If not then consider a need to tune the cf/web server connector. I assume you're using iis. How many sites do you have? And how many connectors (folders in the cf config/wsconfig folder)? What are the settings in their workers.properties file? Are they optimized for the number of sites using that connector?
Also, have you updated cf2018? Are there any errors in the update error log? Did you update the web server connector also?
Are you running the cf2018 pmt (performance monitoring tool set)? Have you updated it?
There could be still more to consider, but let's see how it goes with those. I have blog posts on these and many more topics that would elaborate on things, both at my site (carehart.org) and the Adobe cf portal (coldfusion.adobe.com).
But let's hear if any of this gets you going.
I am trying to connect to an MSTR intelligent server in Seattle from MSTR Developer running on my laptop connected in Bangalore. It takes an average of 10+ seconds for any action I do on the developer, like, login or open folders or open a report or anything. It is almost impractical to do any report development this way (not to mention the frustration).
When my colleague connects to the same instance/project from Seattle he doesn’t face any delays. So I figure that this is a network issue and doesn’t have much to do with the metadata or indexes. The network response time to the box is 30ms and 300ms average from Seattle and Bangalore respectively. I found online that 280ms is average response time from India to US. Accessing the reports and projects via the web interface is smooth though.
Have you ever experienced a situation like this before? Can the network delays cause that much trouble on MicroStrategy? Please help…
PS: This question is not quite a fit for SO. But I guess that MSTR
developers face this problem normally and may be they know a fix.
Hence posting this here rather than SU or somewhere else.
This is a pretty common problem, in my experience. I believe that MicroStrategy's network traffic is XML based, so network bandwidth as well as latency is an issue.
Usually, the web server is more responsive because:
It is performing "simpler" tasks that Developer
The network-intensive traffic is between I-Server and web server, so if they're colocated, performance will be reasonable.
I'm afraid I've never come across an effective solution to this issue. Having a "jump server" in the same data centre as the MSTR servers, with the Developer software installed, is usually the most tolerable solution (provided Remote Desktop isn't too laggy).
Same solution here : we have developpers VMs on a host in the same datacenter as the server, and we remote desktop them. From there, we use Developper/object manager, etc
You can still do 90% of the tasks in web.
If I want to stress test a 'classic' client-server (desktop app <-> LAN <-> database server) Windows Forms desktop application to see how it performs when many concurrent PC users are using it, how should I go about it? I want to simulate many PC users concurrently going through a work flow, to see if it all stands up and at what point the system degrades unacceptably. I've looked at many test tools but they all seems to be skewed toward testing functionality or web app performance, which is quite different.
Clearly having many actual people on actual PCs is not practical, and lots of virtual machines on a few PCs is not representative either. 'Cloud' computing (EC2, Azure etc) looks promising but the documentation and pricing information all seems to be skewed towards mobile apps or web servers, again not the same (but that could just be presentation so I remain open to the idea). I need to be able to virtualise a small LAN of many client machines running the application and a database server.
Can anyone suggest how to do this, or recommend something?
TIA
IMHO the real question is - do you really need to do performance testing in your case? Consider this - where is your business and functional logic?
Performance testing of Desktop applications is oxymoron by itself. Desktop application is made to be used by one person at a time. So if getting a response takes 5 seconds, it will take (pretty much) 5 seconds no matter how many users are clicking the button. The only real thing close to your backend is the DB and they by design support serious asynchronous load. In case this is not enough - just make a cluster.
I'm planning on hosting 3 or 4 WCF/.NET 3.5 REST webservices to be used by a new iPhone application.
I've heard good reviews about DiscountASP.NET (http://www.discountasp.net/index.aspx), but I'm pretty ignorant about shared hosting performance. At the same time, I think it's still early to pay $90 a month for a scalable Amazon EC2 server instance.
So, any idea how many hits/month would a shared hosting website handle?
It depends on how good your shared hosting is. To determine simultaneous users, you can try to benchmark the performance of your server by hitting with many requests.
Your host will likely yell at you if they see you using a lot of your cpu. For web services, bandwidth isn't so much an issue as is your cpu/memory availability.
You should be more concerned about simultaneous users instead of hits per month. You want to be able to handle any spikes in traffic. Shared hosting is less predictable because you don't control the usage of the other users on the machine.
I would say if you're starting out, shared hosting would probably be fine, just monitor it and upgrade when you see decreasing response times. Your host will probably let you know when you are affecting the performance of everyone else on the server.
They are all different. It depends on how much bandwidth they are giving you. (GB/Month is a typical metric).
I would say that hits per month could range in the low hundred thousands without causing major issues on many managed hosting. It all depends how intenst your WCF services are. The advantage is that many managed hosting facilities will allow you to dynamically add more hosting power to your site. (For more $ obviously).
There's alot of information out there on setting up LAMP stacks on a single box, or perhaps moving MySQL onto it's own box, but growing beyond that doesn't seem to be very well documented.
My current web environment is having capacity issues, and so I'm looking for best-practices regarding configuration tuning, determining bottlenecks, security, etc.
I presently host around 400 sites, with a fair need for redundany and security, and so I've grown beyond the single-box solution - but am not at the level of a full ISP or dedicated web-hosting company.
Can anyone point me in the direction of some good expertise on setting up a great apache web-farm with a view to security and future expansion?
My web environment consists of 2 redundant MySQL servers, 2 redundant web-content servers, 2 load balancing front-end apache servers that mount the content via nfs and share apache config and sessions directories between them, and a single "developer's" server which also mounts the web-content via nfs, and contains all the developer accounts.
I'm pretty happy with alot of this setup, but it seems to be choking on the load prematurely.
Thanks!!
--UPDATE--
Turns out the "choking on the load" is related to mod_log_sql, which I use to send my apache logs to a mysql database. By re-configuring the webservers to write their sql statements to a disk file, and then creating a separate process to submit those to the database it allows the webservers to free up their threads much quicker, and handle a much greater load.
You need to be able to identify bottlenecks and test improvements.
To identify bottlenecks, you need to use your system's reporting tools. Some examples:
MySQL has a slow query log.
Linux provides stats like load average, iostat, vmstat, netstat, etc.
Apache has the access log and the server-status page.
Programming languages have profilers, like Pear Benchmark.
Use these tools to identifyy the slowest/biggest offenders and concentrate on them. Try an improvement and measure to see if it actually improves performance.
This becomes a never ending loop for two reasons: there's always something in a complex system that can be faster and as your system grows, different functions will start slowing down.
Based on the description of your system, my first hunch would be disk io and network io on the NFS servers, then I'd look at MySQL query times. I'd also check the performance of the shared sessions.
The schoolbook way of doing it would be to identify the bottlenecks with real empirical data.
Is it the database, apache, network, cpu, memory,io? Do you need more ram, sharding(+), is the DiskIO, the NFS network load, cpu for doing full table scans?
When you find out where the problem is you might run into the problem that its not enough to scale the infrastructure, because of the way the code works, and you end up with the need to either just create more instances of you current setup or make the code different.
I would also recommend as a first step in terms of scalability, off-load your content to a CDN like Edgecast. Use your current two content servers as additional web servers.