We have a web interface for our software using perl and Apache with mod_perl. On a recent Ubuntu 14 installation (Apache 2.4.7, perl version 5.18.2) we've had problems with it randomly stopping working with the error below. This will happen after a random amount of time (eg a few hours or days), has not happened on any previous Ubuntu or CentOS installation, and we've only been able to temporarily resolve it by restarting Apache.
The difficulty in debugging it stems from the fact that it runs fine for a while, processing hundreds or thousands of requests, and we've not been able to identify any particular trigger for it to stop working.
Would anyone have ideas on how to debug and resolve it? Thank you.
The following is the error message. It is repeated with every web request until Apache is restarted. The Utils.pm mentioned is part of our software, and is "use"d in line 2 of index.pl. Utils.pm itself "use"s quite a few other modules.
[Sun Jul 27 19:26:18.110765 2014] [:error] [pid 26316:tid 139927794730752] Attempt to reload Utils.pm aborted.\nCompilation failed in require at /path/to/index.pl line 2.\nBEGIN failed--compilation aborted at /path/to/index.pl line 2.\n
I approach these problems through process of elimination. If a deployment isn't fully successful, I want to know how it differs from fully successful deployments.
I would begin by examining the versions of Perl dependencies. Different distributions might include different versions of non-core modules. If your deployment procedure includes pulling dependencies from CPAN, you may have newer versions of modules on the recent deploy than on previous deploys.
If I found that the dependencies differed, I'd deploy the same versions from an acceptable deployment to the new deployment. If it fixed the problem, I'd know that I have to refine the process for future deployments. If it didn't fix the problem, I'd move on to other identifiable differences between acceptable and unacceptable deployments. "Identifiable" depends on what resources I have at my disposal. I'm an almost-adequate system administrator, so I'd be likely to consult with a colleague to help me identify inconsistencies at that level.
We know that there are environments where the software works as expected. We might not be able to isolate the precise root-cause, but it is reasonable to expect that we can make the environment more hospitable to the application.
The problem seems to be resolved by disabling Apache2::Reload with the following in apache2.conf:
PerlInitHandler Apache2::Reload
PerlSetVar ReloadAll Off
Related
I'm attempting to try out some modifications in SAP's Privileges.app. Unfortunately, their (understandable) Support policy is
This project is 'as-is' with no support, no changes being made. You are welcome to make changes to improve it but we are not available for questions or support of any kind.
Unfortunately, this app uses two constructs I've never come across before in my professional experience, an XPC service + helper (Launch daemon?) and a DockTile plugin. I'm having a hard time just fundamentally getting the app to work when launched from Xcode - it launches, but it seems that there are issues between (maybe?) sandboxing, signing and perhaps entitlements? I've updated the signing to use my own team, of course, and everything compiles/links/launches properly, but when the XPC service tries to install the helper tool it fails
2022-06-29 17:03:56.284544-0500 PrivilegesXPC[13079:128535] [logging-persist] cannot open file at line 45530 of [9ff244ce07]
2022-06-29 17:03:56.284570-0500 PrivilegesXPC[13079:128535] [logging-persist] os_unix.c:45530: (0) open(/var/db/DetachedSignatures) - Undefined error: 0
2022-06-29 17:04:21.060214-0500 PrivilegesXPC[13079:128537] SAPCorp: ERROR! Failed to connect to helper tool: NSCocoaErrorDomain / 4097
2022-06-29 17:04:31.471555-0500 Privileges[13064:127420] SAPCorp: ERROR! Error Domain=NSPOSIXErrorDomain Code=25 "Inappropriate ioctl for device"
2022-06-29 17:04:45.717751-0500 Privileges[13064:129162] SAPCorp: ERROR! Installation of the helper tool failed: Error Domain=CFErrorDomainLaunchd Code=4 "(null)"
As near as I can tell, the last two errors are thrown from a failure in
success = SMJobBless(
kSMDomainSystemLaunchd,
CFSTR("corp.sap.privileges.helper"),
self->_authRef,
&error
);
but I haven't been able to ascertain why this is failing. Searching for errors around Inappropriate ioctl for device has not been fruitful, unfortunately.
If there's anyone out there with some experience in dealing with apps using some of these more esoteric moving parts that can share some things to try, I'd be much obliged. Bonus points if there's any way to debug code running in a DockTile plugin - as near as I can tell, it's running in SystemUIServer, but I can't attach to that (even as root) from Xcode.
I think I've sorted out getting this running. Here's a few roadblocks I encountered.
SMJobBless has some very particular expectations around code-signing - you'll find references to this in some forum posts and there's a sample project that's also referenced with a utility script - which doesn't run on modern macOS because it's written for Python 2 -- which isn't installed by default anymore and a bit difficult to come by. But, after agonizingly converting Python 2-isms over to Python 3, you'll come to find out that that's not the only thing that's changed, a number of the tools (codesign and otool) don't output the same on ARM64 at which time you'll finally stumble across a kind soul that converted SMJobBless.py ... only to find out that it's not actually needed for this project?! Not sure if it's because the Launch Service is contained in the XPC and not the app, but either way - it seems to not be needed.
If you've run Privileges before, it'll have installed it's escalated helper, which will stand in the way of a local Xcode build copying itself over - which matters because of the aforementioned code signing. You'll need to clear away these artifacts
$ sudo rm -rf /Library/PrivilegedHelperTools/corp.sap.privileges.helper
$ sudo rm /Library/LaunchDaemons/corp.sap.privileges.helper.plist
Just deleting them isn't enough, it seems some sort of runtime launchd state needs to be wiped. It's unclear to me if some incantation of launchctl will clear this out, maybe an invocation of launchctl kickstart -k <foo> or something? I ended up rebooting and that seemed to do the trick anyway.
It seems like you need a particular signing certificate to allow the various signing validations that SMJobBless and the XPC communications are doing to be valid. Particularly, it seems you'll need a Developer ID Application, which happens to match what's encoded in the .xcodeproj pulled down from the GitHub repo. This means you can't enable Automatically manage signing as you won't get this type of certificate (as near as I can tell - please correct me if I'm wrong).
Once you've got all that sorted, since you aren't signing with the SAP developer's certificate, your certificate will have a different unique Team ID, so you'll need to update SMAuthorizedClients and SMPrivilegedExecutables, respectively, (look for 7R5ZEU67FQ and replace with your team ID) in
PrivilegesHelper/PrivilegesHelper-Info.plist
PrivilegesXPC/Info.plist
I think that's basically got it. Hope that helps someone else
I have a basic Ubuntu server setup and I am attempting to run a CGI script on it but receiving the following error whenever I try running the script via the command line:
CGI will be removed from the Perl core distribution in the next major release. Please install the separate libcgi-pm-perl package. It is being used at /etc/perl/RABX.pm, line 583.
CGI::Util will be removed from the Perl core distribution in the next major release. Please install the separate libcgi-pm-perl package. It is being used at /usr/share/perl/5.20/CGI.pm, line 29.
Status: 400 Bad Request
Content-Type: application/octet-stream
Content-Length: 98
E1:0,3:513,82:No REQUEST_METHOD in environment; this script must be run in a CGI/FastCGI context,N
If I attempt to run the script in the browser I just get a 500 Internal Server Error.
I'm cautious about installing the libcgi-pm-perl package as the CGI script is part of a much larger legacy application that I'm worried will be incompatible with newer packages. I have no idea if this is a real error or more of a warning though.
The bit that really confuses me is the last line. Any advice would be greatly appreciated!
The full script can be seen here: https://github.com/mysociety/writetothem/blob/master/web/services/queue.cgi
Also, apologies if this question is better suited to ServerFault - I wasn't sure where was more appropriate.
whenever I try running the script via the command line
...
this script must be run in a CGI/FastCGI context
You get the error "No REQUEST_METHOD in environment" because you are not running within a CGI context but instead on the command line.
Please install the separate libcgi-pm-perl package
I'm cautious about installing the libcgi-pm-perl package as the CGI script is part of a much larger legacy application that I'm worried will be incompatible with newer packages. I have no idea if this is a real error or more of a warning though.
This is a warning, that you better install libcgi-pm-perl, because CGI.pm will not be included in core perl in the future. The CGI.pm in this package is the same as it was in core perl, so you don't need to worry about having a different module. Of course, it is probably a newer version than you used before. But this was also the case in the past when you used a new perl version.
its been more than 2 weeks. i try to instal but still getting error.
firstly, indeed I have searched similar error but i didn't find solution at all. if you find it. please let me know.
second, this is my state :
1. i have installed python 2.7 and django 1.5.1 (it works).
2. i also install MAMP.
3. i try to configure mod_wsgi and want it integrated with my MAMP apache server.
4. using mac mountain lion 10.8.4
my configuration file :
/etc/apache2/original/extra/httpd-userdir.conf inside my apache2/original/extra/
/etc/apache2/users/akhyar.conf pastebin.com/zcY58WTV (sorry about this Iam new on stack overflow)
/etc/apache2/httpd.conf pastebin.com/je2D8zMz
third, this is my error :
when i run apachectl configtest this error appears my error
so, what is going on actually? can someone tell me why and show me the mistakes?
if its been solved, what is the next step for configuring mod_wsgi on my MAMP?
thanks before, any help are highly appreciated.
In this file, line 15, you're including the per-user conf files:
http://pastebin.com/7y7ibuqP
On line 473 of this one, one of those per-user conf files, you're including the above file again:
http://pastebin.com/zcY58WTV
This causes infinite recursion while trying to parse the conf files.
I think there are some other errors too, and to be honest the files are pretty messy, but the best way forward is to remove all Include directives from akhyar.conf. For the most part they're already duplicates, where they're not, inline the contents of those files instead of using Include. If there are other errors, you'll at least see useful line numbers to start tracking them down.
Also note that the [warn] lines are just warnings - which you should probably fix, but the server will still run without them, that's not what's causing the error.
I have a few EJBs compiled with Weblogic's EJBC complient with Weblogic 9.2.1.
Our customer uses Weblogic 9.2.3.
During server start Weblogic gives the following message:
<BEA-010087> <The EJB deployment named: YYY.jar is being recompiled within the WebLogic Server. Please consult the server logs if there are any errors. It is also possible to run weblogic.appc as a stand-alone tool to generate the required classes. The generated source files will be placed in .....>
Consequently, server start takes 1.5 hours instead of 20 min. The next server start takes exactly the same time, meaning Weblogic does not cache the products of the recompilation. Needless to say, we cannot recompile all our EJBs to 9.2.3 just for this specific customer, so we need an on-site solution.
My questions are:
1. Is there any way of telling Weblogic to leave those EJB jars as they are and avoid the re-compilation during server start?
2. Can I tell Weblogic to cache the recompiled EJBs to avoid prolonged restarts?
Our current workaround was to write a script that does this recompilation manually before the EAR's creation and deployment (by simply running java weblogic.appc <jar-name>), but we would rather avoid this solution being used in production.
I FIXED this problem by spending a great deal of time researching
and decompiling some classes.I encountered this when migrating from weblogic8 to 10
by this time you might have understood the pain in dealing with oracle weblogic tech support.
unfortunately they did not have a server configuration setting to disable this
You need to do 2 things
Step 1.You if you open the EJB jar files you can see
ejb-jar.xml=3435671213
com.mycompany.myejbs.ejb.DummyEJBService=2691629828
weblogic-ejb-jar.xml=3309609440
WLS_RELEASE_BUILD_VERSION_24=10.0.0.0
you see these hascodes for each of your ejb names.Make these hadcodes zero.
pack the jar file and deploy it on server.
com.mycompany.myejbs.ejb.DummyEJBService=0
weblogic-ejb-jar.xml=0
This is just a Marker file that weblogic.appc keeps in each ejb jar to trigger the recompilation
during server boot up.i automated this process of making these hadcodes to zero.
This hashcodes remain the same for each ejb even if you execute appc for more than once
if you add a new EJB class or delete a class those entries are added to this marker file
Note 1:
how to get this file?
if you open domains/yourdomain/servers/yourServerName/cache/EJBCompilerCache/XXXXXXXXX
you will see this file for each ejb.weblogic makes the hashcodes to zero after it recompiles
Note 2:
When you generate EJB using appc.generate them to a exploded directory using -output C:\myejb
instead of C:\myejb.jar.This way you can play around with the marker file
Step2.
Also you need a PATCH from weblogic.When you install the patch you see some message like this
"PATH CRXXXXXX installed successfully.Eliminate EJB recomilation for appc".
i dont remember the patch number but you can request weblogic for that.
You need to use both steps to fix the problem.The patch fixes only part of the problem
Goodluck!!
cheers
raj
the Marker file in EJBs is WL_GENERATED
Just to update the solution we went with - eventually we opted to recompile the EJBs once at the Customer's site instead of messing with the EJBs' internal markers (we don't want Oracle saying they cannot support problems derived from this scenario).
We created two KSH scripts - the first iterates over all the EJB jars, copies them to a temp dir and then re-compiles them in parallel by running several instances of the 2nd script which does only one thing: java -Drecompiler=yes -cp $CLASSPATH weblogic.appc $1 (With error handling of course :))
This solution reduced compilation time from 70min to 15min. After this we re-create the EAR file and redeploy it with the new EJBs. We do this once per several UAT environment creations, so we save quite a lot of time here (55min X num of envs per drop X num of drops)
I've been trying to get RApache set up properly on my Macbook Pro running OS X 10.5.8. After installing RApache successfully (I think), I added the following to the httpd.conf file as suggested in the manual.
LoadModule R_module /apache/module/path/mod_R.so
ROutputErrors
<Location /RApacheInfo>
SetHandler r-info
</Location>
With these additions, I was then able to successfully go to http://localhost/RApacheInfo and see the status information. If I add the following additional line,
REvalOnStartup "library(brew)"
I can no longer load the RApacheInfo page successfully.
I don't know what the issue is here. Without the REvalOnStartup call, I'm seeing the following in the Apache error log.
[Fri Feb 26 11:36:36 2010] [notice] Apache/2.2.13 (Unix) mod_ssl/2.2.13 OpenSSL/0.9.7l DAV/2 mod_R/1.1.8 R/2.10.1 configured -- resuming normal operations
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
Not a good sign. Any thoughts on what might be going on? Or things to check?
Chris
I'm not sure why you're getting this error on just loading a library, but we've seen this issue on Leopard due to Apple's freetype/fontconfig/cairo libraries calling CoreFoundation calls:
http://finzi.psych.upenn.edu/R/Rhelp02/archive/118681.html
How did you install R?
I tried reproducing your issue on Snow Leopard with stock apache2, binary R 2.10.1 from CRAN, Xcode 3.2.1, and rapche 1.1.8 but no luck. My setup worked like a champ.
I don't have access to Leopard anymore, so my best suggestion is to use gdb to break on that really long system call. You will want to remove the REvalonStartup directive and replace it with a handler call which does the same, configure apache to spawn only one child process (hence the whole fork thing) and connect gdb to it. Then point your web browser to your handler url and see if gdb breaks on the function and then look at the stack trace.... Pretty lame I know, but that's all I got for now.