I am running a Flask app on an Apache 2.4 server. The app sends requests to an API built by a colleague using the Requests library. The requests are in a specific format and constructed by data stored in a MySQL database. The site is designed to show the feedback from the API on the index, and the user can edit the data stored in the MySQL database (and by extension, the data sent in the request) by another page, the editing page.
So let's say for example a custom field date is set to be "2006", I would access the index page, a request would be sent, the API does its magic and sends back data relevant to 2006. If I then went and changed the date to "2007" then the new field is saved in MySQL and upon navigating back to index the new request is constructed, sent off and data for 2007 should be returned.
Unfortunately that's not happening.
My when I change details on my editing page they are definitely stored to the database, but when I navigate back to the index the request sends the previous set of data. I think that Apache is causing the problem because of two reasons:
When I reset the server (service apache2 restart) the data sent back is the 'proper' data, even though I haven't touched the database. That is, the index is initially requesting 2006 data, I change it to request 2007 data, it still requests 2006 data, I restart the server, refresh the index and only then does it request 2007 data like it should have been doing since I edited it.
When I run this on my local Flask development server, navigating to the index page after editing an entry immediately returns the right result - it feeds off the same database and is essentially identical to the deployed server except that it's not running on apache.
Is there a way that Apache could be caching requests or something? I can't figure out why the server would keep sending old requests until I restart it.
EDIT:
The requests themselves are large and ungainly and the responses would return data that I'm not comfortable with making available for examples for privacy reasons.
I am almost certain that Apache is the issue because as previously stated, the Flask development server has no issues with returning the correct dataset. I have also written some requests to run through Postman, and these also return the data as requested, so the request structure must be fine. The only difference I can see between the local Flask app and the deployed one is Apache, and given that restarting the Apache server 'updates' the requests until the data is changed again, I think that it's quite clearly doing something untoward.
Dirn was completely right, it turned out not to be an Apache issue at all. It was SQL Alchemy all along.
I imagine that SQL Alchemy knows not to do any 'caching' when it requests data on the development server but decides that it's a good idea in production, which makes perfect sense really. It was not using the committed data on every search, which is why restarting the Apache server fixed it because it also reset the connection.
I guess that's what dirn meant by "How are you loading data in your application?" I had assumed that since I turned off Flask's debugging on the development server it would behave just like it would in deployment but it looks like something has slipped through.
Related
I have a question about using AFNetworking with SDURLCache.
In my application. I can see they work together fine, for repeated queries, responses from local cache will be used correctly.
But, let's say the server xml is updated one minute ago, in my app, cache is still used.
This doesn't match my client's expectation, the client's expectation is:
I need to check http header, if it is 200, load the file from server, if it is 304 (not modified), just use local cache from disk/
Any chance this is a built-in behaviour? Or I have to manually hack the class somewhere so that it can behave like this?
I'm building a web tool which allows users to upload PDFs to a server using their web browsers. The server is based on Python (Paste + Pyramid).
The problem I have right now is the following: If a user uploads a rather large file (let's say 100 MB) and they cancel the upload before it is completed, my handler code on the server is still called (instead of the request being aborted).
The problem is that the request.POST['myfile'].file is incomplete when that happens. This effectively means that the PDF file is corrupted if I simply write it to some place on the server.
When I watch the server's log, it shows a "broken pipe" exception within the Paste server; however I have no idea how to catch that exception and have it prevent my view/handler code from executing and storing the incomplete file.
Seems like the paster HTTP server does not correctly validate the uploaded form data and simply passes the request down the WSGI pipeline even if the connection (HTTP POST) was closed by the user.
I worked around this issue by simply setting up NGINX to act as a reverse proxy. This also adds some security benefits as it might be better tested than paster.
Update:
My main problem was that I was using runserver (the built in web server of manage.py). After some trial and error we ended up using WSGI.
More specifically, uWSGI and Nginx as web server. Static content is served directly by Nginx while dynamic pages are piped through uWSGI and are handled by the Python web app.
Unless you are doing something fancy (like tracking the upload progress, etc), your pylons controller should not be invoked until the entire file has been uploaded.
Is there one? Apparently not. The H2 automatic mixed mode is described
here.
Revival for further reference.
As stated by #fredt, as far as I know, there is no official magic parameter to achieve mixed mode.
Still, you can always start a server programmatically using a Server object so that other process are able to connect to your database.
I discovery a trick to accomplish something pretty close to mixed mode. In order to do that you will need to set the remote_open property to true and connect using this form of URL.
The idea here is doing something like this:
Try connecting to the server using the kind of URL stated above.
If you can't connect it means the server haven't been started, so go ahead and start the server programmatically.
When you connect again, one of three things will happen.
If no database file exists, one will be created in the specified file path and the server will start serving it from the URL alias.
If the database file exists and it isn't being served, the server will open the file from the specified path and start serving it.
If the database file exists and it is being served, the server will simply return a connection.
I'm not sure if it is safe to use that kind of pattern when you plan to spawn a lot of short lived processes (particularly I haven't dive into HSQLDB code to check how it handles database / creation / opening for multiple simultaneous requests when remote_open is set). Still, I've been using this kind of pattern to share development databases between Web Applications for a while and never ran into a single database corruption problem.
The main limitation here is that when the application acting as a server is closed, open connections will stop working and throw Exceptions... Which is not an Issue for my development environment, here this will generally only means one or two broken requests until another server is started and the connection pool detects and renews its connections.
No HSQLDB does not support such a mode.
We have a client who has just recently acquired their first SharePoint 2010 system. They have two servers, one running SQL Server 2008 R2, and the other one is used to run SharePoint 2010. When I restore a backup of the site collection from our development environment the server with running SharePoint, the site works fine for about a day. After that day, when we try to connect to the home page of the site, the site battles to connect. The browser just says “Connecting…” until the page eventually times out. But you can still access the “backend” pages, like the View all Site Content page (http:///_layouts/viewlsts.aspx) and the site settings page (http:///_layouts/settings.aspx).
Here’s a little info about the web parts that we are running on the home page. The web parts checks if the cache is empty, and if the cache is empty it populates the cache with all the items in a specific list. The list contains a lot of items, +/- 4000 items. So it’s obvious that SharePoint will be retrieving a lot of data from the SQL Server.
If I delete the web applications (that includes deleting the content database and the IIS web site) and I re-deploy the site collection using the backup I’ve made on our development environment the site works fine again for about a day.
After running into these problems we started monitoring the resources use on the SQL Server (Using the Resource Monitor). If you filter the network traffic to only display the network usage of the sqlserver.exe process, it shows that it’s only communicating at +/- 30KB/s. This is incredibly slow! When you copy a 390MB file from the SharePoint Server to the SQL (to test the connection speed between the two servers) it copies the file in 2 seconds.
This is a very strange problem that raises a couple questions. First of all our development environment is almost exactly the same as their environment (in fact, we have less RAM) so why don’t we have any problems like this in our development environment? Secondly, why if you deploy the site from scratch, does the site work for a day, and only the start causing problems later? And the final question: why is the communication speed between the SharePoint and SQL so slow, but the connection speed between the two servers is very quick?
I really hope someone can help us with this, or give us a couple of things we can troubleshoot.
Thanks in advance!
After a very long struggle we found a solution.
We found this post:
http://trycatch.be/blogs/tom/archive/2009/04/22/never-turn-off-quot-auto-create-amp-auto-update-statistics-quot.aspx
We tested it and it worked!!!
So all we had to do was switch "Auto create statistics" and "Auto update statistics" to true, and to problem was solved
Thanks for all the replys
"why is the communication speed
between the SharePoint and SQL so
slow, but the connection speed between
the two servers is very quick?"
You've shown its not the network with the file copy test so that would indicate that either
a) SQL server is overloaded and can't keep up or
b) the WFE is overloaded and can't keep up.
Have you performed basic performance troubleshoting looking at both servers for things like CPU/Memory/Disk swapping etc etc
Also there are lots of specific steps to troubleshoot SQL performance. This reference is for 2005 but all the same principals apply.
I work on quite a few DotNetNuke sites, and occasionally (I haven't figured out the common factor yet), when I use the Database Publishing Wizard from Microsoft to create scripts for the site I've created on my Dev server, after running the scripts at the host (usually GoDaddy.com), and uploading the site files, I get an error... I'm 99.9% sure that it's not file related, so not sure where to begin in the DB. Unfortunately with DotNetNuke you don't get the YSOD, but a generic error, with no real way to find the actual exception that has occured.
I'm just curious if anyone has had similar deployment issues using the Database Publishing Wizard, and if so, how they overcame them? I own the RedGate toolset, but some hosts like GoDaddy don't allow you to direct connect to their servers...
The Database Publishing Wizard's generated scripts usually need to be tweaked since it sometimes gets the order wrong of table/procedure creation when dealing with constraints. What I do is first backup the database, then run the script, and if I get an error, I move that query to the end of the script. Continue restoring the database and running the script until it works.
There are two areas that I would look at -
Are you running in the dbo schema and was your scripted database
using dbo?
Are you using an objectqualifier in either your dev or your
production environment? (look at your sqldataprovider configuration
settings)
You should be able to expose the underlying error message by setting the following in the web.config:
customErrors mode="Off"
Could you elaborate on "and uploading the site files"? New instance of DNN? updating an existing site? upgrading DNN version? If upgrade or update -- what files are you adding/overwriting?
Also, when using GoDaddy, can you check to verify that the web site's identity (network service or asp.net machine account depending on your IIS version) has sufficient permissions to the website's file system? It should have modify permissions and these may need to be reapplied if you are overwriting files.
IIS6 (XP, Server 2000, 2003) = ASP.Net Machine Account
IIS7 (Vista, Server 2008) = Network Service
Test your generated scripts on a new local database (using the free SQL Express product or the full meal deal). If it runs fine locally, then you can be confident that it will run elsewhere, all things being equal.
If it bombs when you run it locally, use the process of elimination and work your way through the script execution to find the offending code.
My hunch is that the order of scripts could be off. I think I've had that happen before with the database publishing wizard.
Just read your follow up. In every case that I've had your problem, it was always something to do with the connection string in web.config. Even after hours of staring at it, it was always a connection string issue in web.config. Get up, take a walk and then come back.
If you are getting one of DNN's error pages, there is a chance it may have logged the error to the eventlog table.
Depending on exactly what is happening and what DNN is showing you you might be able to manually look inside the EventLog table, pull out the XML data stored there, and parse it to find the stack trace and detailed information regarding the specific error at hand.
I have found however though that I get MUCH better overall experiences with deployments using backups and restores of my database, that way I am 100% sure that all objects moved correctly, and honestly it works better in my experience.
With GoDaddy I know another MAJOR common issue is incorrect file permissions, preventing DNN from modifying the web.config and other files that it needs to do.