We are currently testing an upgrade from CF11 to CF2018 for my company's intranet. To give you an idea how long this site has been running, our first version of CF was 3.1! It is still using application.cfm, and there is code from 1998, when I started writing this thing. Yes, 21 years -- I'm astonished, too. It is a hodgepodge of all kinds of older frameworks, too, including Fusebox.
Anyway, we're running Win 2012 VM connected to a SQL 2016 farm. Everything looked OK initially, but in the Week I've been testing, the server has come to a slowdown once (a page took more than 5 seconds to run, something that usually takes 100ms, no DB involvement), and another time, the server came to a grinding halt. The only way I could restart CF App service was by connecting to the server with another server via Services, because doing it via Remote Desktop was so slow.
Now keep in mind -- it's just me testing. This is a site that doesn't have a ton of users, but still, having 5 concurrent connections is normal and there are upwards of 200-400 users hitting this thing every day.
I have FusionReactor running on this thing now, so the next time a lockup happens, I will be able to take a closer look, but what do you think is the best way I can test this? Our site is mostly transactional, users going and filling out forms to put internal orders through. We also connect to XML web services and REST services; we also provide REST services, too. Obviously there's no way to completely replicate a production server's requests onto a test server, but I need to do more thorough testing. Any advice would be hugely appreciated.
I realize your focus for now is trying to recreate the problem on test. That may not be as easy as hoped. Instead, you should be able to understand and resolve it in production. FusionReactor can help, but the answer may well be in the cf logs.
You don't mention assessing the logs at the time of the hangup. See especially the coldfusion-error log, for outofmemory conditions.
You mention raising the heap, but the problem may be with the metaspace instead. If so, consider simply removing the maxmetaspace setting in the jvm args. That may be the sole and likely cause of such new and unexpected outages.
Or if it's not, and there's nothing in the logs at the time, THEN do consider FR. Does IT show anything happening at the time?
If not then consider a need to tune the cf/web server connector. I assume you're using iis. How many sites do you have? And how many connectors (folders in the cf config/wsconfig folder)? What are the settings in their workers.properties file? Are they optimized for the number of sites using that connector?
Also, have you updated cf2018? Are there any errors in the update error log? Did you update the web server connector also?
Are you running the cf2018 pmt (performance monitoring tool set)? Have you updated it?
There could be still more to consider, but let's see how it goes with those. I have blog posts on these and many more topics that would elaborate on things, both at my site (carehart.org) and the Adobe cf portal (coldfusion.adobe.com).
But let's hear if any of this gets you going.
Related
First let me say i am only a novice programmer, and by no means an sql guru. We have an app at work that is and has been under heavy dev from the vendor for sometime (2+ years). It runs as a MSSQL instance on one of our servers, and there is a client install for the desktops. The client software is making direct sql calls to the database.(it also has a local mysql instance to handle the client settings) there is 6-12 ports that had to be opened up for the communication. Looking at the sql manager, i can see direct sql calls from various clients.
Seems to me this is entirely the wrong approach. the closest thing i have done to this, was a webpage + php+ mysql. The webpage would make requests, and all the processing would be serverside, then simply display the results. The sluggishness my users feel i think is from the clientside request+ processing of the sql data.
ps: i realize that if they have not done it by now, switching to another paradigm seems out of the question. i just want to know if i am way off base.
You are way off base.
The client side has much more processing power.
Consider the case of one server and 5 clients. Even is the server has 3 times the power of a client the clients as a whole are still 5:3 more powerful.
If the application is sluggish it was probably poorly written. You need to investigate the root cause. Client / Server is a leading practice in design, I'm guessing it is not the root cause. It might be badly implemented or there might be other reasons. Your comment about having a local mysql sounds very fishy to me -- there should be no need for this.
I have a .NET application (VB.NET) that runs against a MS Access database. Every data request connects to the access database, runs and returns the query and closes the connection back again.
I placed the database on a windows xp 32-bit machine.
I have two clients on which I installed the .NET application. Both clients are running windows 7 professional 32-bit.
Now I have a performance problem with this.
When I use the first client it runs fine. All data is shown very fast. When I than use the second client, it takes some 10 seconds to connect to the database, fetch the data and close the database connection. When i ask for other data on that second client, it all runs fine, until I request data from the first client than back again. Than it takes again 10 seconds on the first client before my data is fetched.
Can anybody please help me with that? I owe a Belgian beer to the solver of this issue ;-)
Thanks!
Tom Wickerath wrote a great article on improving multiuser performance for MS Access applications. While his article assumes a MS Access front-end, many of the tips should apply to a .Net application. I recall two points that might help you:
Keep a persistent connection to the back-end
Use (short) UNC paths instead of mapped drives
After a long search, i found it out... My virusscanner NOD32 was causing this, most probably by excessive scanning inbound and outbound network traffic.
I'm not sure stackoverflow is the right place for questions like this, but ...
It sounds like the first process is locking the file, so the second process has to wait.
"Use SQL Server" isn't a completely flippant response - SQL Server is specifically designed to handle concurrency issues like this.
IMHO ...
PS:
This is a pretty lame link, but it might help:
http://office.microsoft.com/en-us/access-help/about-sharing-an-access-database-on-a-network-mdb-HP005240860.aspx
PPS:
Here's a somewhat better link, with some suggestions for things you can do to improve concurrency:
http://www.softcoded.com/web_design/upgrading_access.php
I am just wondering why is restarting IIS 7 application pool consider a good practice? I know that it'll clean up orphan resources like threads or session state, but how does other web server deal with this kind of problem? Do java developer/admin have to restart weblogic/websphere/apache/tomcat every so often to clean up junk in memory?
Typically it's used to compensate for bad programming. Sometimes yours, sometimes third parties depending on the libraries you are using.
And, as with all recommendations, it's best to look at your specific environment to see if it's really necessary before doing it.
Start with profiling the app. A simple way is to just let it run for a while under load. Look for things like does memory utilization settle down or does it continually grow?
If it keeps growing, you will need to set IIS to recycle every so often and go fix your app. If it grows to a point then settles down then you should be just fine. Make sure you test even the edge cases of your app.
With IIS it is possible to have non-managed code - the best example is a COM object - that don't behave well. As a result you can set your IIS to recycle processes periodically, to allow for the memory leakage that might occur with such code.
It's not necessary to do it though. You can eliminate the restart thresholds, if you prefer not to restart the processes.
If this idea doesn't exist in other servers, then it may be because other servers allow only managed code.
Our website seems to be slower than it used to be, how can I test that? And is there a way to find the cause? (eg too many visitors).
Thanks.
There is a rather good tool for performance benchmarking of web servers: Jakarta Jmeter, which is an Apache project, so it's rather well supported and tested.
The key to be able to pinpoint the cause would be to do benchmarking regularly, so you can actually match changes in your benchmark results with events on your server: upgrades, code changes, variations in the number of visitors...
The Firebug add on for Firefox has a Net tab which is useful for debugging issues and testing. Also Fiddler on Windows is nice. And then there is the age old tradition of checking your server error logs for any problems.
A good first step is to make sure you are keeping fairly complete server logs and feed them into a log analyser. This is helpful for giving you a general idea of how long things take and which pages are slowest. It's also a good idea to check your error logs to make sure things are working properly.
Beyond that, things get more complicated as you may need to isolate your webserver, code and database to see if one of these is the bottleneck. Also, Jeff's blog, coding horror had a recent entry on server optimization.
Use Google Analytics to track your site's visitors over time to find out if you are getting more traffic.
You tagged your question with shared-hosting - being on a shared host means that someone else's code running on the same machine as your may be affecting your site's performance.
I'd suggest going with Varkhan and apphacker's suggestion to make sure your site's code is reasonably quick. use Analytics to get some stats and the possibly, depending on how many visitors you are getting and how slow the site is, consider moving away from a shared host.
Try the server speed checker at Bitcatcha.com. The tool ping your website server and record the time needed to get a response. It also pings from 8 different nodes to your server. You are able to at least find out whether it's your server that is slowing your website.
I'm attempting to help our network engineers troubleshoot a situation for one of our clients. This client purchased a point-of-sale system from quite literally a "mom-and-pop" vendor, and said vendor recommended SQL Server Express 2005 as the back-end database to save the client from having to incur extra licensing fees. (Please don't get me started on that!)
We didn't write the app, and because it's a commercial app, we have no source code available. (Not that it would help us if we did; the thing was built in PowerBuilder, so we don't have tooling for it.) The app does none of its own logging, that we can ascertain. All we have to go on is SQL Server Express's own logging.
In the application, an end user swipes a membership card. Occasionally (a few times a day), the swipe will not return data from the database. The message on screen will say, "Member 123 not found." (The member numbers are actually six digits, "000123.") A rescan immediately afterward returns the member data correctly.
We've eliminated the scanner itself as a source of issues -- it routinely scans the full six-digit number. A scan of SQL Server Express's log indicates that it is coming back online from being idle, often at the point of the scan (but also at several other times per day). (Idle mode is explained here.)
I understand that allocating/deallocating RAM the way SQL Express does is a time-consuming process, especially if we're talking about hundreds of megabytes at a time -- which appears to be the case.
What we're not sure of is whether or not we're getting back partial data, or if the app is simply failing to connect to the database and displaying a generic error message. Since everything is so opaque, and the client is (for obvious reasons) unwilling to pay us to sit in their facility for 8 hours or so to physically see it happen (perhaps with network monitoring/packet sniffing tools), we're kind of at a loss.
At this point, our recommendation is that the client upgrade to SQL Server 2005 Workgroup Edition, with 5 CALs. But that doesn't completely sit well with me as the solution to this issue, because I'm reasonably certain that no SQL Server ever returns partial data -- if you can't connect, you can't connect. (That said, I still recommend it because it's a solution to a number of their other issues!)
I don't have much experience with Express. (I never use it for anything but local development, and there only at home; I certainly never recommend it to my clients.)
My question to those who might have experience with Express is, have you ever seen an instance of SQL Express return partial data, without the app itself being the cause of it? Specifically, have you seen this behavior when returning from idle mode?
(For what it's worth, we're inclined to believe that the app is failing to connect and merely displaying a generic error message, lopping off leading zeroes on the member ID when it does. That seems the most reasonable answer -- a third question might be, do you guys concur with that assessment?)
I've never heard of or experienced SQL Server Express returning partial data. It's essentially the same code base as the full SQL Server.
It is more likely that the application is experiencing a timeout (which defaults to 30 seconds) due to SQL Server Express going idle. The application probably receives a timeout that it does not expect and does not handle it well.
The problem and possible solutions are discussed in this forum thread: http://social.msdn.microsoft.com/forums/en-US/sqlexpress/thread/a8fbf8d6-9949-47a5-a32b-50f8131f1127/
I suspect you have a connection string that looks like this:
Data Source=.\SQLEXPRESS; Integrated Security=True;AttachDbFilename=|DataDirectory|\myDatabase.mdf;User Instance=True
From the referenced thread:
This connection string will cause an
initial connection to the main
instance (.\SQLEXPRESS) and then
instruct the main instance to spawn a
new instance of SQL Server under the
user's context and attach the database
specified to that new User Instance.
The User Instance is a completely
separate running instance of SQL
Server form the main instance that is
unique to the user and that will be
shut down when there are no longer any
connections to it.
This is totally different that
attaching a database to the main
instance, which stays running at all
times, unless you've manually shut it
down. If your question is about the
main instance going into an Idle
state, then your question is not
unique to SQL Express and you should
ask this question in the Database
Engine forum. I believe all Editions
of SQL Server have an Idle state and
the other forum would be where you can
find out how to affect that behavior.