RavenDB database restore: Operation failed: Cannot access file, the file is locked or in use - ravendb

A year ago, when we started using RavenDB, we quickly hit this error in production:
Operation failed: Cannot access file, the file is locked or in use
We found out through some forums, that we could get rid of this by not running RavenDB as an IIS website, but as a service. We did, and never saw the error again.
Until now: I'm setting up a new environment and thought, I'd give it a whirl with running RavenDB off IIS, but the error quickly reappeared.
Facts and what I've tried:
running off v.30155 on a commercial license
using a custom AppPool (called RavenApplicationPool)
have data-folder separate from IIS dir (through Raven/WorkingDir AppSetting)
RavenApplicationPool user has "Full control" permissions on both IIS folder and data-folder
Windows Authentication installed and enabled for website
overlapped recycle disabled for app-pool
webdav-publishing not installed (as requested in setup docs)
hitting my keyboard (no luck)
The error occurs at random when site has been running for a few minutes. It also occurs when trying to restore a backup using the following command:
C:\RavenDBExecutables\Server\raven.Server.exe --restore-source=c:\SOME_PATH\my-db-backup.raven --restore-database-name=my-db --restore-database=http://localhost:8080
(the RavenDBExecutables folder is merely the zip of binaries/executables et.c. - it does not overlap with the IIS website folder)
When running the restore, these folders are created:
IndexDefinitions (contains a lot of files ending in .index)
Indexes (lot of folders with integer names)
logs (empty)
system (empty)
temp (empty)
root folder (my-db) is empty, i.e. no Data file, .resource.database or raven-data.ico as with working db's
After failed import, if I visit http://localhost:8080/docs/Raven/Restore/Status, I get
*a lot of lines like "Copying PATH_TO_INDEX_FILE", *
"Esent Restore: Failure! Could not restore database!",
"Microsoft.Isam.Esent.Interop.EsentFileAccessDeniedException: Cannot access file, the file is locked or in use\r\n at
Microsoft.Isam.Esent.Interop.Api.Check(Int32 err)\r\n at
Microsoft.Isam.Esent.Interop.Api.JetRestoreInstance(JET_INSTANCE
instance, String source, String destination, JET_PFNSTATUS
statusCallback)\r\n at
Raven.Database.Storage.Esent.Backup.RestoreOperation.Execute()",
"Unable to restore database Cannot access file, the file is locked or in use"
What can it be then?

The reason this happens is that IIS by default does overlapped recycle.
That means that both versions of the app are running (and both are trying to use the same resources).
With overlapped recycled set to false, that shouldn't happen.
We have seen similar errors when users have set two databases to the same path

Related

NotesSQL to Notes 9 ODBC Connection resulting in IM005(0)[Microsoft][ODBC Driver Manager] Driver's SQLAllocHandle on SQL_HANDLE_DBC failed

Every application I use to initiate an ODBC connection that I've created to my IBM Notes 9 DB results in getting an error during connect:
Driver's SQLAllocHandle on SQL_HANDLE_DBC failed
I have scoured the web trying to correct this and have not been able to find an answer that solves for this issue on my machine. I am able to open and access the Lotus Notes DB within the Notes 9 software. I am able to create the ODBC System DSN and I know that it's connecting to the server properly because all of the available .nsf files populate in the "Database:" dropdown menu when going through the new ODBC connection setup. I am able to see my username which gets populated from my Notes ID file. The issue occurs when I use something to initiate the ODBC connection. I have tried Excel, QlikView, AQT, Teradata SQL Assistant all with the same result. Once I choose the ODBC connection that I've created, and click on "Ok" in each application I've attempted this with, I am met with the same error above.
I used Event Viewer per a suggestion in an older post online and received the message below:
Could not load NSQLE32.EXE. This file must exist in the same directory NSQL32.DLL and NSQLV32.DLL is in. It is possible NSQLE32.EXE could not be loaded because NotesSQL couldn't locate a valid Notes/Domino installation (couldn't find NNOTES.DLL) -- this may be because user (MYUSERID) does not have the correct rights to the Notes/Domino directory. It could also be because NotesSQL can not find your Notes/Domino installation in the Registry. NotesSQL looks for the NNOTES.DLL file by looking in the path pointed to by the following Registry entries -- HKEY_LOCAL_MACHINE\Software\Lotus\Notes{version}\Path : HKEY_LOCAL_MACHINE\Software\Lotus\Domino{version}\Path. If neither of these registry entries exist or they point to an invalid version of Notes/Domino NotesSQL will not work. To resolve -- please re-install Notes/Domino.
Some notes (no pun intended) and things I've done:
Verified that the DLLs mentioned in Event Viewer do exist in the proper directories.
Run the Nsql_ALM.exe application to configure the NotesSQL driver.
Used the 32-bit ODBC Administrator because my Notes 9 is 32 bit, as well as the driver. The 64 bit driver would not even allow installation (attempted this after hitting this roadblock) but the 32 bit installed successfully.
Modified my System Environment Variable for Path to include the true location to notes.ini, which resides in:
C:\Users\MYUSERID\AppData\Local\Lotus\Notes\Data
Modified my System Environment Variable for Path to include the location to the NotesSQL driver, which resides in:
C:\NotesSQL
Modified my registry to ensure that the appropriate strings exist to reference the proper files:
HKEY_CURRENT_USER\Software\Lotus\Notes\NotesIniPath (to notes.ini)
HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\Notes\9.0\Path (to notes application)
HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\Notes\9.0\DataPath (to notes.ini)
HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\Domino\9.0\Path (to notes application)
HKEY_LOCAL_MACHINE\SOFTWARE\Lotus\Domino\9.0\DataPath (to notes.ini)
Set the NSQLE32.exe application to run in compatibility mode with all other options.
Attempted with everything run as administrator.
Uninstalled both Notes and NotesSQL and reinstalled everything cleanly again.
Attempted a log via Tracing within ODBC Administrator and it will not create a log file when I am attempting the connection. It will populate, however, with all of the system ODBC connections when the calling application attempts a lookup of the existing connections to populate in the dropdown menu. If I start tracing after the dropdown menu has been populated, then attempted the ODBC connection, even multiple times and with multiple versions, a new log file is never created.
I'm not sure where to go from here. Has anybody had this issue and is there something else that I can do in order to fix it?
I was able to correct the issue based on information in the Event Logs (Windows - Application) indicating that the Notes.INI file could not be located. I first added the location of notes.ini to the user and system environment variables, however, the issue persisted. I then moved notes.ini to the c:\NotesSQL folder where database connectivity dll's such as NSQL32.dll are found. This corrected the issue.
The application calling notes is SAP BusinessObjects Edge (Enterprise - Crystal Reports) 4.2 SP07.
notes.ini can not be found - Windows application event log

Windows could not start the Apache CouchDB service on Local Computer

I have installed CouchDB on my Windows machine but while starting the CouchDB service, I am getting a message like:
Windows could not start the Apache CouchDB service on Local Computer. The service did not return an error. This could be an internal Windows error or an internal service error. If the problem persists, please contact your system administrator.
As the service is not running, I am unable to access Fauxton too.
I am using Windows 7. CouchDB is 2.0.0. Port 5984 is not in use.
I don't think your question is a duplicate of https://stackoverflow.com/a/44107335/219187 because you are on Windows 7, and the problem described there is for Windows 10 with the creators update.
But maybe the solution fixes your problem as well? Here is the procedure:
Download the prelease build 2.2.4-101 from https://nssm.cc/download
Stop the CouchDB service through the Windows Services dialog (paused is not enough)
Overwrite nssm.exe in <CouchDbInstallDir>\bin with the one from the downloaded ZIP file (make sure you pick the right version 32 bit / 64 bit)
Start the CouchDB service
Issue it's happening since the last updates released by Microsoft. I'm not completely aware of what's causing it, but I think it's something related to CouchDB service not been able to start using Local Administrator rights.
However I've managed to start the service manually, by doing so:
Open Command Prompt - in the Search from the Start Menu or Task Bar type "cmd"
Run it as an Administrator - right click on the Command Prompt application and choose "Run as administrator" option /this is really IMPORTANT as it will allow the service to have administrator access/
Navigate to the folder where CouchDB is install - default path is "D:/CouchDB", but could be anywhere else; you have to find it
Go to the "bin" folder in there
Type "couchdb" as a command to start the service
You will see a message showing after this - "kernel-poll not supported; "K" parameter ignored"
If it adds some error messages after it or closes the whole terminal, you're making some things bad from this guide, so follow it strictly.
You can now open up the Fauxton application in the browser like normal from here - http://localhost:5984/_utils/
Keep in mind that you have to leave the cmd opened in order the service to be working as expected. As far as I saw no information was lost, so it's all good.
This is a temporary solution though, as we are waiting a relase from either Microsoft or Apache to solve the issue, or at least give us more explanation about it.
i just met the same problem.
the cause is space, you have to install CouchDB in a path without any space, even Program Files folder, because there is a space between Program and Files...

What permissions/policies are needed to support loadUserProfile=true for new application pools?

Something happened on my development workstation (Windows 8.1) in the last few weeks which require me to either run my App Pools with the "Load User Profile" setting at False or not run with the identity set to ApplicationPoolIdentity. If I were to create a new app pool, using ApplicationPoolIdentity as the identity and with loadUserProfile=true, the following happens when trying to load the application in a browser:
A number of errors in the Windows Event Log (both System and Application types):
Warning event 1509 - Windows cannot copy file \\?\C:\Users\Default\AppData\Local\Microsoft\VSCommon\12.0\SQM\sqmdata-7236-039-00000.sqm to location \\?\C:\Users\[Name of App Pool]\AppData\Local\Microsoft\VSCommon\12.0\SQM\sqmdata-7236-039-00000.sqm. This error may be caused by network problems or insufficient security rights.
Error event 1511 - Windows cannot find the local profile and is logging you on with a temporary profile. Changes you make to this profile will be lost when you log off.
Another 1509 warning
Error event 1500 - Windows cannot log you on because your profile cannot be loaded. Check that you are connected to the network, and that your network is functioning correctly. DETAIL - Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
5 x event 5022 warnings - The Windows Process Activation Service failed to create a worker process for the application pool '[App Pool Name]'. The data field contains the error number.
Finally an error 5002 - Application pool '[App Pool Name]' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
The App Pool is shut down, as the error 5002 said
"HTTP Error 503. The service is unavailable." is then seen in the browser. Any further requests are met with the same (which makes sense since the app pool is shut off).
I've seen a common "fix" for this here and here which basically say to turn off profile loading. Yes it makes the problem go away, but this doesn't get to the root cause. I know that it is possible to run with this configuration as a I have a Windows 2012 machine which supports the configuration just fine. In this case, hitting an app with a new app pool set to ApplicationPoolIdentity and loadUserProfile=true actually creates the new user profile (I can watch as the profiles folder is created in C:\Users) and the app runs merrily. What's worse is I know this configuration worked on the problem machine just a few weeks ago. I have a number of App Pools I created which have their own profiles and folder under the C:\Users folder. These app pools work just fine NOW with the ApplicationPoolIdentity and loadUserProfile=true settings. It's just that NEW app pools refuse to run and load a user profile.
Does anyone have any insight to what might be going on?
Edit: I read the bottom of this recent article. It's a bit contradictory in saying that the setting can be turned on, but also says:
Only the standard application pools (DefaultAppPool and Classic .NET AppPool) have user profiles on disk. No user profile is created if the Administrator creates a new application pool.
However, if you want, you can configure IIS application pools to load the user profile by setting the LoadUserProfile attribute to "true".
I'm very confused.
The SQM file listed in the event log warning was created by a Windows or Visual Studio update. When the user profile service or application pool runs and tries to create a new profile, it tries to copy the file to the profile. The SQM file requires administrator permissions to copy. The user profile service or application pool does not have sufficient permissions to copy the file, an error is generated, and the user profile is not created. Without a user profile, the application pool cannot run because it doesn't have an isolated secure place to store data.
Remove or delete the SQM file from the source directory, and the user profile will be created successfully when the app pool is initialized. You can also change the permissions on the SQM file, but I'm not sure what the appropriate permissions should be. The user profile service runs as "LocalSystem Account". See its documentation for permission info. It's unclear to me whether the application pool identity itself is being used to perform the copy operation, or the local system account.
If you remove the file from the source directory, you could also manually copy the file where it was trying to go as well.
After a very brief search about what SQM is, it seems like it is traditionally used as "service quality management". Usually it would contain information to send back to the program authors with metrics, logs, or somesuch. I don't know if this is the case with this file or not. So it doesn't seem like it's important to include it in the new profile.
I can't take 100% credit for this answer, as I was tipped off by a comment attached to an answer on some other question. I can't find the link to it in the 50 browser tabs open for troubleshooting this. That guy deserves a thank you, because I believe this is a much better solution than compromising the security of a server by pooling all the resources together like in IIS 6.
P.S. As noted in your comment, a bug report has been filed.

Server 2012 IIS site on different drive

I'm trying to get IIS 8 on a brand new Server 2012 up and running to run a .net 4.5 app and I can't get it to work across drives
Error: 500.19
Module
IIS Web Core
Notification
Unknown
Handler
Not yet determined
Error Code
0x80070005
Config Error
Cannot read configuration file due to insufficient permissions
Config File
\\?\D:\callsheet-test\web.config
The current site is just a single file index.html.
If I copy the folder to anywhere on c: like c:\callsheet-test\ or c:\inetpub\callsheet-test it works perfectly once I change the target location in IIS 8 (site > basic settings).
It doesn't matter if there is an web.config file as a complex mvc/webforms app gives the same error.
I have tried copying the file with permissions and even thought the permissions seem to match for all users. I am still getting this error.
I have deleted the partition and and recreated it as a mount point c:\mount and the same issue happens. Copy the files to c:\notamount and they work fine.
This server is running on a VMware server. Windows 2012 x64 Standard.
Based on further investigation of the problem (via link in my comment above), I've found a workaround via here: change the Group Policy value of "Audit Removable Storage" from "Not Configured" to "No Auditing" and reboot. Bizarre I know. This setting is in Windows Settings/Security Settings/Advanced Audit Policy Configuration/System Audit Policies/Object Access.
Even though we're using VMWare and not Hyper-V this workaround still worked for me. Hopefully Microsoft can provide a answer.
I also had the same error and the solution was similar. I had installed Sql Server 2012 on Windows Server 2012 and it was at this time that I noticed IIS worker processes running from the system C: drive were unable to access configuration files on the E: drive. The site could only be loaded when on the C: drive, despite all the necessary NTFS permissions being granted through IIS_IUSRS, IUSR, Users and/or even the specific AppPool itself. This became evident when IIS couldn't access the E: drive even when running the AppPool (.NET 4.5) using an admin account other than the ApplicationPoolIdentity account (IIS APPPOOL.NET v4.5 in this context - verified in procexp.exe). Installing Sql Server 2012 was throwing errors during installation and required enabling auditing for object access among other settings.
Solution:
Run -> secpol.msc
Security Settings -> Local Policies -> Audit Policy
Change "Audit Object Access" policy to "No Auditing"
Hopefully, this background helps ease someone's headache. :-)

SQLExpress connection fails in IIS 7 w/ user instance error - "Failed to generate a user instance

Mainly looking to answer my question #1 below, but more knowledge would be appreciated.
I tried to use these resources during my investigation, but was unsuccessful:
http://social.msdn.microsoft.com/Forums/en/sqldatabaseengine/thread/f5eb164d-9774-4864-ae05-cac99740949b (For this error: Failed to generate a user instance of SQL Server due to failure in retrieving the user's local application data path. Please make sure the user has a local user profile on the computer. The connection will be closed.)
http://social.msdn.microsoft.com/forums/en-US/sqlexpress/thread/6dfdcc22-7a81-4e8f-a947-c1ce6982d4b3/ (For this error: CREATE DATABASE permission denied in database master. An attempt to attach an auto-named database for file ? failed. A database with the same name exists, or specified file cannot be opened, or it is located on UNC share.)
Questions
1.) Why does this error occur while running the Telerik Rad Controls for ASP.NET AJAX "Live Demos" project with IIS 7 (Running Telerik Live Demos works fine using ASP.NET Development Server with this connection string)
Failed to generate a user instance of SQL Server due to failure in retrieving the user's local application data path. Please make sure the user has a local user profile on the computer. The connection will be closed.
2.) How is creating a SQL Server Express instances different in IIS 7, from ASP.NET Development Server & SSMSE
3.) Are there certain attributes of a SQL connection string not allowed when running a website on different contexts (based on #2).
Environment:
I'm not running the "Live Demos" .NET 3.5 ASP.NET web application via the ASP.NET Development Server (feature that pops up in your system tray and picks a port for you after clicking play in Visual Studio). That works just fine! I'm running the website on IIS 7. SQL Server Express is using the NETWORK SERVICE user in Control Panel > Administrative Tools > Services > SQL Server (SQLExpress).
Using this connection string provided with the installed "Live Demos" web application demo project:
<add name="NorthwindConnectionString"
connectionString="Data Source=.\SQLEXPRESS;AttachDbFilename=|DataDirectory|Northwind.mdf;Integrated Security=True;User Instance=True"
providerName="System.Data.SqlClient" />
I've tried setting "User Instance=False", but that just throws another error:
CREATE DATABASE permission denied in database master. An attempt to attach an auto-named database for file ? failed. A database with the same name exists, or specified file cannot be opened, or it is located on UNC share.
(where "?" is the path of the *.mdf file - C:\Users\\MyDocuments\Visual Studio 2008\Projects\TelerikDemos\Telerik\RadControls for ASP.NET AJAX Q2 2011\Live Demos\App_Data\Northwind.mdf .. Stack Overflow italics is broken with some of those characters, so I had to remove that path)
Someone answered me on a previous question to set this "User Instance=False", but it appears User instances have nothing to do with whether or not you use SQL Express. User Instances are simply a feature of SQL Express that allows a very unprivileged user to host a database instance in it's own user context.
Note, this Northwind database is stored in an *.mdf file in the App_Data folder (under the "Live Demos" root application directory) along with the *.ldf (log file). I did previously try attaching the *.mdf files as actual databases under the "Databases" folder (in the SSMSE Object Explorer tree), but later removed them.
Web application "Live Demos" root folder (and nested folders/files) have the following users assigned with ALL privileges:
- IIS APPPOOL\Telerik ("Telerik" is the name of my application pool in IIS 7 for this site)
- IUSR
- NETWORK SERVICE
Making a note for myself about this SQLExpress master database query:
SELECT * FROM sys.dm_os_child_instances
Also tried different combinations of *.mdf & *.ldf permissions while also changing the user on the SQL Server (SQLExpress) Windows 7 service (Control Panel > Administrative Tools > Services) .. and also restarted the service after making those changes.
To reproduce:
download the Telerik Rad Controls for ASP.NET AJAX. Set the permimssions I mentioned in the "Live Demos" folder under Program Files\Telerik, change the .NET version of the web application to .NET 3.5, switch out their 3.5 web.config file with the normal web.config file in that folder. You have to use Visual Studio 2010, but I am running this in Visual Studio 2008 (with a little grunt work I did because our company is not yet on VS2010). Also switch out the proper Bin35 assemblies into the "Live Demos" folder Bin folder. Compile the solution. Create an IIS 7 website. Add Windows authentication. Enabled anonymous and Windows authentication.. all others are disabled. Set application pool to use Classic and 32 bit.
Then navigating to this URL and clicking the "First Look" image.
http://localhost/combobox/examples/overview/defaultcs.aspx
====================
More evidence will be provided if requested.
You are using a connection string with trusted authentication = true. This means that the connection uses the security context of the calling process.
When you run with the development server you are running in the security context of the logged in user, so every thing works fine.
When you run in IIS you are in the security context of the application pool process, which is NETWORK SERVICE, which does not have a user profile, therefore it crashes.
You can fix it by either:
Change the identity of the application pool to a normal user with access to the database
Use a connection string with user name and password
IIS doesn't load the Windows user profile, but certain applications might take advantage of it anyway to store temporary data. SQL Express is an example of an application that does this. However, a user profile has to be created to store temporary data in either the profile directory or in the registry hive. The user profile for the Network Service account was created by the system and was always available. However, with the switch to unique Application Pool identities, no user profile is created by the system. Only the standard application pools (DefaultAppPool and Classic .NET AppPool) have user profiles on disk. No user profile is created if the Administrator creates a new application pool.
However, if you want, you can configure IIS application pools to load the user profile by setting the LoadUserProfile attribute to "true".
https://learn.microsoft.com/en-us/iis/manage/configuring-security/application-pool-identities