We have a DotNetNuke site running on two servers that are load balanced. To ensure the files are in sync on these servers, we are using File Replication Service.
Search works fine on DotNetNuke when not load balanced, but in the load balanced setup the search stops working after a while (no suggestions, no results).
The following related exception is all over our log files:
[D:2][T:31][ERROR] DotNetNuke.Services.Exceptions.Exceptions - Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: NativeFSLock#D:\Sites\SiteName\App_Data\Search\write.lock
at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit)
at Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, MaxFieldLength mfl)
at DotNetNuke.Services.Search.Internals.LuceneControllerImpl.get_Writer()
at DotNetNuke.Services.Search.Internals.LuceneControllerImpl.Delete(Query query)
at DotNetNuke.Services.Search.Internals.InternalSearchControllerImpl.DeleteSearchDocumentInternal(SearchDocument searchDocument, Boolean autoCommit)
at DotNetNuke.Services.Search.Internals.InternalSearchControllerImpl.DeleteSearchDocumentsByModule(Int32 portalId, Int32 moduleId, Int32 moduleDefId)
at DotNetNuke.Services.Search.SearchDataStore.StoreSearchItems(SearchItemInfoCollection searchItems)
at DotNetNuke.Services.Search.SearchEngine.IndexContent()
at DotNetNuke.Services.Search.SearchEngineScheduler.DoWork()
My best guess is that the issue is caused because both servers are running their search functionality, and the File Replication Service is syncing the files which causes conflicts.
What would be the best way to solve this?
Add an exclusion rule to not replicate the search index folder, but let both servers keep running search?
Somehow disable one server from indexing?
Any other suggestions?
Installation details:
DNN v. 09.02.00 (366)
.NET Framework 4.6
There's a 'scheduler' tool inside of the settings section that contains all CRON/background jobs functionality.
One of the background jobs is the 'Search: Site Crawler' job which is responsible for indexing the website. When that job runs at the same time on both servers, unexpected conflicts occur. To prevent this from happening, you can configure the job to only run on a specified server using the 'Servers' setting.
After configuring the job to only run on one server, the issue did not come back and search still works on both servers.
Thanks #Sanjay for pointing me in the right direction.
If I remember correctly search is done via a scheduled task. Have you tried setting up the task to run on only one server and then use file replication to sync across to the other server.
Related
with a new project we encountered some strange behaviour on our ColdFusion application.
Whenever a single request is initiated from the browser, the code of the cfml-templates is
executed multiple times. Upon viewing the corresponding log-files we found out, that indeed
for some reason the same request fires the evaluation in our application multiple times. One request
generates several entries. This is especially the case for long-running requests, such as database imports.
The ColdFusion application implements a REST-service, but even on manually requesting a resource,
such as a certain cfml page, on the same application - the code gets executed an unknown amount of times(variable initializations, database write-operations etc. take place), and if the request runs too long (cap at around ~4-6 seconds) there is no response to the browser.
About the infrastructure:
The application is Coldfusion18 with Tomcat Standard Edition
The webserver is an Apache (2.4.6).
Everything runs on a Linux machine with Cent OS 7.7
The corresponding Java version is 11.0.4
Our best guess is that there might be some misscommunication between the coldfusion connector with
the apache webserver. We actually searched for some configuration parameters that could cause the
problem, without success. Upon an installation on a windows machine we did not encounter that error.
Anyone got any idea?
we just found our answer in the following post:
Link to Solution
I am attempting to host a Saga from one project in another project using NServiceBus 6 with SqlPersistence and SqlDialect.MsSqlServer. In most examples I have found, the Saga is contained in the same assembly as the hosting app, and perhaps this is why I am struggling.
When hosting everything in the same app, the NServiceBus.Persistence.Sql.MsBuild package correctly outputs Saga .sql files during the build and then picks these up and executes them on run. Using a separate app, only the Outbox, Subscription and Timeout .sql files are generated, not the Saga ones. The following entry is then logged on run:
INFO NServiceBus.Persistence.Sql.Installer Directory '[PATH]\SagaPersistence\Service\bin\Debug\NServiceBus.Persistence.Sql\MsSqlServer\Sagas' not found so no saga creation scripts will be executed.
A full VS 2017 repro may be found at https://github.com/WolfyUK/NServiceBusSagaSqlPersistence.
Firstly, is it a bad idea to host a Sagas from another service, rather than being self-hosted? If not, can someone advise the best way to resolve the SQL Persistence issue?
Can you add NServiceBus.Persistence.Sql.MsBuild to the Saga project? The scripts should then be found there. Unfortunately they're not copied to the host its folder, so you'll have to take them from there into production. Or have them generated by using EnableInstallers, like you're already doing.
Our users are restless. They keep complaining about woolly, unmeasurable stuff, particularly slowness, without giving specifics, which of course makes it very difficult to track down.
Nonetheless, it is quite possible that they are right, that there are server calls that are taking way too long to come back. So I want to put some kind of sniffer on the web site (we're using ASP.NET MVC 4 on IIS7) that will log any call that takes more than n seconds to turn around, or that returns more than x megabytes of data, along with all request parameters, the response size, and maybe a certain amount of response data.
I haven't a clue how to do this, though. Any suggestions?
here is my take on this:
FRT
While you can use failed request tracing to log slow requests, in my experience is more useful for finding out why a request fails before it hits your application, rather than why its running slowly. 9/10 times its going to simply show you that the slowdown is in your code somewhere.
Log Parser
Yes you can download and analyze iis logs. I use Log Parser Lizard to do the analysis - its a great gui over log parser. Here's a sample of how you might query slow requests over 1000ms:
SELECT
To_String(To_timestamp(date, time), 'dd/MM/yyyy hh:mm:ss') As Time,
cs-uri-stem, cs-uri-query, cs-method, time-taken, cs-bytes, sc-status
FROM
'C:\inetpub\logs\LogFiles\W3SVC1\u_ex140721.log'
WHERE
time-taken > 1000
ORDER BY time-taken desc
New Relic
My recommendation - go easy on yourself and sign up for a free trial. No I don't work for them, but I've used their APM product a lot. Install the agent on the server - set it up. In 10 mins you will be amazed at the data you see about the site. Trust me.
Its designed to work in production environments and gives you amazing depth of info on what's running slow, down to the database query and stack traces. Its pure awesome. Once its setup wait for the next user complaint, log in and look at traces for the time frame.
When your pro trial ends, you can still get valuable data on the free tier, but it will only keep last 24 hours. We purchased licenses -expensive yes, but worth every cent. Why? Time taken to identify root causes was reduced by an order of magnitude, we can get proactive by looking at what is number 2, 3 and 4 on the slow requests list and working those before they become big problems, and finally the alerting makes us much more responsive when things were going wrong.
Code it
You could roll you own. This blog uses Mvc ActionFilters to do the logging. You could also use an HttpModule similar to this post. The nice thing about this approach is you can compile and implement the module separately from your application, and then just drop in the dll and update web.config to wire up the module. I would be wary of these approaches for a very busy site. Also, getting the right level of detail to fully identify the root is challenging.
View Requests
As touched on by Appleman1234, IIS has a little known feature to look at requests currently executing. Its handy for the 'hey its running slow right now' situation. You can use appcmd.exe or the IIS gui to do it. You will need to install the 'Request Monitor' IIS feature for this to work. This approach is ok for rudimentary narrowing of the problem, but does not show you whats running slowly in your controller.
There are various ways you can do this:
Failed Requests Tracing(FRT) – formerly known as Failed Request Event Buffering (FREB) with custom failure condition of takes over a certain time to load / run
Logging request information with IIS logging functionality and then using a tool like LogParserStudio
Using tools like Fiddler or IISMonitor on the IIS server to capture request information
For FRT the official documentation is available here and information how to capture dumps for long running process is avaliable here
For logging request information in IIS information about log file analysis is located here
For information on configuring Fiddler to capture IIS requests find information here
A summary of the steps in the linked resources is provided below.
For FRT
From IIS Manager for a given site,In the Actions pane, under Configure, click Failed Request Tracing and enter desired values in dialog box to enable Failed Request Tracing.
From IIS Manager for a given site, under IIS click Failed Request Tracing Rules, in order to define rules of failure for a given request. In the Actions pane, click Add and follow the wizard.
The logs will go in the directory you specify and are viewable in a web broswer.
For IIS logging
Logging is enabled by default on IIS
From IIS Manager for a given site,under IIS click Logging, and in the Actions Pane, click Enable to enable logging if it isn't already.
From IIS Manager for a given site,under IIS click Logging, and then configure as desired and click apply.
Install LogParser, .Net 4.x and LogParserStudio (if you need additional steps see here
Open LogParserStudio and add logs to it, you then can use SQL queries to get information from the log files.
For Fiddler
You need to change the user that IIS runs as to a user that can launch applications, like Fiddler (instead of Network Service), and then launch Fiddler with that user.
Also see Monitor Activity on a Web Server (IIS 7) for further information.
I want to deploy my WCF web Service on Azure plateform.
I have created a Storage account for my website, and also created a cloud Service and uploaded my package file and config file to the staging site.
But while uploading, The message displays
'Your staging deployment is starting. Hang on, the page will refresh once the deployment begins.'
I am waiting sice 2-3 hours and not getting the desired output.
Am I doing correctly? Or is there anything that I forgot?
Please Help...!
Most likely there is a problem in your code or the packaging that is causing the role to continuously restart. This is a fairly common problem, but there are a lot of possible causes (missing an assembly reference, an uncaught exception, the Run() method is exiting, a Startup Task is failing, or many other things). You need to gather more information to know exactly what the problem is and how to fix it.
There are many threads here on SO about this topic. There's also a Microsoft post discussing how to diagnose this type of issue. Those are good places to start.
When running a web deploy to a specific IIS site I get the following error:
Error: The synchronization is being stopped because the maximum number of sync passes '5' has been exceeded even though all the changes could not be applied. This could occur if there are external changes being made to the destination.
At C:\Code\.....\deploy.ps1:185 char:10
+ & <<<< ($appDeployCmd) $type /M:$url /U:$user /P:$pass /A:Basic -allowUntrusted -useCheckSum
+ CategoryInfo : NotSpecified: (Error: The sync...he destination.:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Web Deploy is working fine on this environment against other IIS sites and file syncs are also working. I have previously been able to use web deploy to deploy this specific site without issue. All of the sudden out of nowhere, this issue started happening and I can no longer deploy this site.
I'm doing a basic site deploy with a package built from msbuild. I don't think the specifics are that important because as I said this was all working before and currently works against other sites on the same server farm without issues.
The error message says:
"This could occur if there are external changes being made to the destination."
but I'm not sure how to track this down or if it is even the issue to begin with. I've made sure all explorer windows are closed in all remote sessions. I've tried restarting the site and the app pool. The only thing I have not tried is rebooting the server which is not possible at moment.
Any ideas what might be cause this web deploy to fail?
I had the same error and the problem was my dropbox.
I was working directly in my dropbox folder, and when you publish, it causes dropbox to syncronize at the same time, which caused the error.
Disabling dropbox sync while working solved the problem.
I recon the problem also could happen with onedrive, google drive and so on.
We had this problem when converting from a previously adhoc deploy of a service to MSDeploy, and found that if there were files that were either
marked as read-only via the DOS/Windows read-only file attribute.
inaccessible due to ACLs
then we would get the "maximum number of sync passes" error on deploying.
Once we fixed the attributes/ACLs, we were able to sync.
Quick and easy way to resolve this issue is to delete the files in the destination and re-run the web deploy.
The issue seems to revolve around the ACL step of the web deploy, which attempts to change the permissions of your websites files as a safety measure intended to ensure they are not changed during a deployment.
By default Web Deploy sets the ACL of the sites anonymous user to read only while also overwriting Control Panel access to your website.
Source
You can turn of ACL in future to avoid this if you wish, but it's not really worth it. This will also speed up web deploys - but that is a separate issue.
Not really an answer, but one workaround you can try if you are using the Web Deploy dirPath, filePath, or contentPath providers is the ignoreErrors provider setting. If you know that you are consistently hitting a certain error number, you can specify that that error be ignored when it's hit. See the dirPath provider article for full details (and caveats).
In my case I couldn't fix it but realised the deployment worked regardless.
If you are reading this I wouldn't suggest to just assume it worked, and if it did that it deployed fully, but consider that it may be a false alarm!