I have been consolidating 3 Databases into one via prefixes in my mediawiki installation. I got three wikis using the same database like so:
en_interwiki
de_interwiki
es_interwiki
Everything works fine out of visitor perspective... but whenever a USER wants to post a new article or commit edits, the database takes up to 35 seconds to respond. This is unacceptable.
I activated debugging like so:
# Debugging:
$wgDBerrorLog = '/var/log/mediawiki/WikiDBerror.log';
$wgShowSQLErrors = true;
$wgDebugDumpSql = true;
$wgDebugLogFile = '/var/log/mediawiki/WikiDebug.log';
$wgShowDBErrorBacktrace = true;
I am getting debug info, and it seems that pagelinks is the culprit, but i am not one hundred percent sure.
Did anyone ever have this issue before?
Please help me!
Best regards,
Max
I could fix it. In my case, the memcache had the wrong port. Everything is back to normal.
In case anyone uses memcache with their MediaWiki installation: Be sure to use the right port on your server, or you will end up like me, with 30 second-wait-times.
Related
Is there a way to make the logged user (on superset) to make the queries on impala?
I tried to enable the "Impersonate the logged on user" option on Databases but with no success because all the queries run on impala with superset user.
I'm trying to achieve the same! This will not completely answer this question since it does not still work but I want to share my research in order to maybe help another soul that is trying to use this instrument outside very basic use cases.
I went deep in the code and I found out that impersonation is not implemented for Impala. So you cannot achieve this from the UI. I found out this PR https://github.com/apache/superset/pull/4699 that for whatever reason was never merged into the codebase and tried to copy&paste code in my Superset version (1.1.0) but it didn't work. Adding some logs I can see that the configuration with the impersonation is updated, but then the actual Impala query is with the user I used to start the process.
As you can imagine, I am a complete noob at this. However I found out that the impersonation thing happens when you create a cursor and there is a constructor parameter in which you can pass the impersonation configuration.
I managed to correctly (at least to my understanding) implement impersonation for the SQL lab part.
In the sql_lab.py class you have to add in the execute_sql_statements method the following lines
with closing(engine.raw_connection()) as conn:
# closing the connection closes the cursor as well
cursor = conn.cursor(**database.cursor_kwargs)
where cursor_kwargs is defined in db_engine_specs/impala.py as the following
#classmethod
def get_configuration_for_impersonation(cls, uri, impersonate_user, username):
logger.info(
'Passing Impala execution_options.cursor_configuration for impersonation')
return {'execution_options': {
'cursor_configuration': {'impala.doas.user': username}}}
#classmethod
def get_cursor_configuration_for_impersonation(cls, uri, impersonate_user,
username):
logger.debug('Passing Impala cursor configuration for impersonation')
return {'configuration': {'impala.doas.user': username}}
Finally, in models/core.py you have to add the following bit in the get_sqla_engine def
params = extra.get("engine_params", {}) # that was already there just for you to find out the line
self.cursor_kwargs = self.db_engine_spec.get_cursor_configuration_for_impersonation(
str(url), self.impersonate_user, effective_username) # this is the line I added
...
params.update(self.get_encrypted_extra()) # already there
#new stuff
configuration = {}
configuration.update(
self.db_engine_spec.get_configuration_for_impersonation(
str(url),
self.impersonate_user,
effective_username))
if configuration:
params.update(configuration)
As you can see I just shamelessy pasted the code from the PR. However this kind of works only for the SQL lab as I already said. For the dashboards there is an entirely different way of querying Impala that I did not still find out.
This means that queries for the dashboards are handled in a different way and there isn't something like this
with closing(engine.raw_connection()) as conn:
# closing the connection closes the cursor as well
cursor = conn.cursor(**database.cursor_kwargs)
My gut (and debugging) feeling is that you need to first understand the sqlalchemy part and extend a new ImpalaEngine class that uses a custom cursor with the impersonation conf. Or something like that, however it is not simple (if we want to call this simple) as the sql_lab part. So, the trick is to find out where the query is executed and create a cursor with the impersonation configuration. Easy, isnt'it ?
I hope that this could shed some light to you and the others that have this issue. Let me know if you did find out another way to solve this issue, or if this comment was useful.
Update: something really useful
A colleague of mine succesfully implemented impersonation with impala without touching any superset related, but instead working directly with the impyla lib. A PR was open with the code to change. You can apply the patch directly in the impyla src used by superset. You have to edit both dbapi.py and hiveserver2.py.
As a reminder: we are still testing this and we do not know if it works with different accounts using the same superset instance.
I am not able to create connection between metabase and LDAP.
I am trying to set it up similar with our Zeppelin configuration:
ldapRealm.contextFactory.url = our_url<br>
ldapRealm.contextFactory.systemUsername = our_username<br>
ldapRealm.contextFactory.systemPassword = our_password
ldapRealm.userDnTemplate = cn={0},ou=people,dc=xha,dc=app<br>
ldapRealm.memberAttributeValueTemplate = cn={0},ou=people,dc=xha,dc=app<br>
ldapRealm.authorizationEnabled = true<br>
ldapRealm.searchBase = dc=xha,dc=app<br>
ldapRealm.userSearchBase = ou=people,dc=xha,dc=app<br>
ldapRealm.groupSearchBase = ou=group,dc=xha,dc=app<br>
ldapRealm.userSearchAttributeName = uid<br>
ldapRealm.userSearchFilter = (&(objectclass=*)(uid={0}))<br>
ldapRealm.memberAttribute = memberUid
When I tried similar configuration:
URL, Username, Password, worked well.
I tried use “ou=people,dc=xha,dc=app” for “USER SEARCH BASE”, but it did not work and I get error in log:
errors {:ldap-user-base "User search base does not exist or is
unreadable"}}
So it seems, that “USER SEARCH BASE” is in reality “GROUP SEARCH BASE”.
The only one value which working in “USER SEARCH BASE” is “OU=group,DC=xha, DC=app”
I tried for “USER FILTER”:<br>
cn={login},ou=people,dc=xha,dc=app<br>
(&(objectClass=)(uid={0}))<br>
(&(objectClass=)(uid={login}))<br>
(&(objectClass=)(memberUid={0}))<br>
(&(objectClass=)(memberUid={login}))<br>
{uid={login}}<br>
(&(objectclass=*)(cn={login}))<br>
I also tried with empty filter.
Nothing works yet and I am stuck on this for like 2 weeks.
In log is just:
{:errors {:password “did not match stored password”}}
I also tried steps in https://discourse.metabase.com/t/ldap-debugging/2741 but I am still not able to make it work.
Our “LDAP” accounts have no email -> https://discourse.metabase.com/t/autenticate-by-ldap-a-user-that-have-no-email/5215
I do not know if this is still “hardcoded” in version: 0.34.2
So, do you have any suggestions, how can I figured it out?
Thank you in advance
Peter
Actually I figured it out. "OU=people,DC=xha, DC=app" was correct, but our LDAP schema was huge and there were errors in LDAP's logs, that "SIZE LIMIT EXCEEDED". So we started using MariaDB for storing Metabase settings and I saved settings with "OU=group,DC=xha, DC=app" and then manually changed to OU=people,DC=xha, DC=appin MariaDB.
I have found that all data of my 5 YEARS old site tables was suddenly mixed up
some data that cannot be updated via any existing sp is updated.
After long search of sp i came to conclusion that somebody messing with my site.
I assume that its done via sql injections.
I have huge amount of trafic on my site 24/7 ,site has more than 100 pages, and the logs are now just showing what user entered what page...more logs will slow down the site even more. so now i need to act efficiently.
1.What is the best way to find where someone injecting
2.how to log his ip and time of injection
never done this before, read lots of mixed opinions on google. please advice your best practise.
Instead of tracking down the "bad guys" you should focus on restoring your database and making your code resistant or invulnerable to injections, not sure whats the best way for asp.net but in java it is well known that prepared statements make it impossible to have somebody peform a sql injection on your data.
Check out this link for how to improve your code in asp.net:
Classic ASP SQL Injection Protection
This is some code I use on my page as a "catch all" attempt for injection attempts via query strings (ie, data sent through a URL):
trap = 0
ref = lcase(request.querystring)
if ref <> "" then
badChars = array("OR 1=1", "select", "drop", "shutdown", "--", "insert", "delete", "waitfor", "union", "0x31", "convert", "truncate", "sysobjects")
cn = 0
for i = 0 to uBound(badChars)
if instr(ref,badchars(i)) > 1 then cn=cn+1
next
if cn >=2 then trap = 1
end if
if trap = 1 then .... ban user ip code here
You could simply put "if trap = 1 then response.end" which would stop any further action on the page. I prefer to ban the IP for an hour.
This should also work with request.form for form input.
You may also want to sanitize your variables that take form input.
data=request.form("emailaddress")
data = replace(data,"'","")
data = replace(data,"union","")
etc.
We have just changed our domain after protracted name change (the name actually happened two years ago!) and our DokuWiki installation has stopped being able to see any groups and memberships.
The config has been updated to reflect the new server and DCs and login is working correctly, it is only the groups that aren't working.
$conf['auth']['ldap']['server'] = 'ldap://MYDC.mydomain.co.uk:389';
$conf['auth']['ldap']['binddn'] = '%{user}#mydomain.co.uk';
$conf['auth']['ldap']['usertree'] = 'dc=mydomain,dc=co,dc=uk';
$conf['auth']['ldap']['userfilter'] = '(userPrincipalName=%{user}#mydomain.co.uk)';
$conf['auth']['ldap']['mapping']['name'] = 'displayname';
$conf['auth']['ldap']['mapping']['grps'] = 'array(\'memberof\' => \'/CN=(.+?),/i\')';
$conf['auth']['ldap']['grouptree'] = 'dc=mydomain,dc=co,dc=uk';
$conf['auth']['ldap']['groupfilter'] = '(&(cn=*)(Member=%{dn})(objectClass=group))';
$conf['auth']['ldap']['referrals'] = '0';
$conf['auth']['ldap']['version'] = '3';
$conf['auth']['ldap']['debug'] = 1;
Obviously I have edited the doain name there, but for the life of me I can't see what's wrong here, It all worked fine yesterday on the old domain.
I should also state that this is an old version of DokuWiki that for various reasons I can't actually update.
The debug line gives me a "ldap search: success" line, but if I add "?do=check" onto any url within the system I get "You are part of the groups"...... and nothing, it can't see any groups.
It's a massive pain as we have a pretty intricate ACL setup for the site, so it's not like I can just throw it open to all.
If anyone has any suggestions, no matter how obvious, please pass them on.
Solved it by changing the dokuwiki authentication plugin that was used, the 'authad' is more simple to use and just works with what I'm doing.
As a side bonus it also means that I have finally been able to get the install upgraded to the current version.
we've got a real confusing problem. We're trying to test an SQL Bulk Load using a little app we've written that passes in the datafile XML, the schema, and the SQL database connection string.
It's a very straight-forward app, here's the main part of the code:
SQLXMLBULKLOADLib.SQLXMLBulkLoad4Class objBL = new SQLXMLBULKLOADLib.SQLXMLBulkLoad4Class();
objBL.ConnectionString = "provider=sqloledb;Data Source=SERVER\\SERVER; Database=Main;User Id=Username;Password=password;";
objBL.BulkLoad = true;
objBL.CheckConstraints = true;
objBL.ErrorLogFile = "error.xml";
objBL.KeepIdentity = false;
objBL.Execute("schema.xml", "data.xml");
As you can see, it's very simple but we're getting the following error from the library we're passing this stuff to: Interop.SQLXMLBULKLOADLib.dll.
The message reads:
Failure: Attempted to read or write protected memory. This is often an indication that other memory has been corrupted
We have no idea what's causing it or what it even means.
Before this we first had an error because SQLXML4.0 wasn't installed, so that was easy to fix. Then there was an error because it couldn't connect to the database (wrong connection string) - fixed. Now there's this and we are just baffled.
Thanks for any help. We're really scratching our heads!
I am not familiar with this particular utility (Interop.SQLXMLBULKLOADLib.dll), but have you checked that your XML validates to its schema .xsd file? Perhaps the dll could have issues with loading the xml data file into memory structures if it is invalid?
I try to understand your problem ,but i have more doubt in that,
If u have time try access the below link ,i think it will definitely useful for you
link text
I know I did something that raised this error message once, but (as often happens) the problem ended up having nothing to do with the error message. Not much help, alas.
Some troubleshooting ideas: try to determine the actual SQL command being generated and submitted by the application to SQL Server (SQL Profiler should help here), and run it as "close" to the database as possible--from within SSMS, using SQLCMD, direct BCP call, whatever is appropriate. Detailing all tests you make and the results you get may help.