We have Oracle Unified Directory(OUD) with around 10 million users. We have read-only account that is used to for search operations on base dn. But sometimes this user account is being used to perform wildcard search like (uid=*) on base dn. This is causing timeout issues on OUD and impacting performance also. Is there a way to restrict this wildcard search on base dn?
You can set resource limited on the user that is performing the wildcard search.
This will of course cause all the queries they run to be limited to the amount of entries set and have unknown consequences in whatever app is using the returned data. It’s probably preferable to educate them on how to perform paged searches so you can still define limits and they can still consume the data.
Create an ldif file and run ldapmodify with limits suited for your environment.
dn: uid=baduser,ou=people,dc=example,dc=com
changetype: modify
add: ds-rlim-lookthrough-limit
ds-rlim-lookthrough-limit: 10000
-
add: ds-rlim-size-limit
ds-rlim-size-limit: 5000
-
add: ds-rlim-time-limit
ds-rlim-time-limit: 300
Related
I can accomplish this with memberOf but due to API constraints with Okta, this goes over our rate limit and is too slow for production use. I am trying to do this with uniqueMember as this supports indexing but I am having a real hard time getting this to query correctly.
Here is what I have and I believe the issue is how I am thinking of uniqueMember. I believe it is just an attribute for the groups but I don't know how to query a list of groups to get the uniqueMembers.
(&(objectClass=inetOrgPerson)(|(uniqueMember=CN=groupnamehere,ou=groups,dc=myorg,dc=okta,dc=com)
The memberof query that works
(&(objectClass=inetOrgPerson)(|(memberOf=CN=group1,ou=groups,dc=myorg,dc=okta,dc=com)
Assume I have an array of N DNs (distinguished names), e.g.:
cn=foo,dc=capmon,dc=lan
cn=bar,dc=capmon,dc=lan
cn=Fred Flintstone,ou=CapMon,dc=capmon,dc=lan
cn=Clark Kent,ou=yada,ou=whatnot,dc=capmon,dc=lan
They are not related and I cannot reduce/simplify the search. I have N complete DNs and want N records.
Can I write a single LDAP search that will return exactly N records, one for each DN? The assumption being that performance of both client and server will be better if I do it all in one search. Had it been SQL, it would be:
SELECT *
FROM dc=capmon,dc=lan
WHERE dn IN (
"cn=foo,dc=capmon,dc=lan",
"cn=bar,dc=capmon,dc=lan",
"cn=Fred Flintstone,ou=CapMon,dc=capmon,dc=lan",
"cn=Clark Kent,ou=yada,ou=whatnot,dc=capmon,dc=lan"
)
rather than doing individual LDAP searches in a for loop (which I do know how to do).
I tried against an MS Active Directory. There, all fields (seem to) have a distinguishedName attribute, and a search filter like this works (I added some newlines for readability):
(|
(distinguishedName=cn=ppolicy,dc=capmon,dc=lan)
(distinguishedName=cn=Users,dc=capmon,dc=lan)
<more ORed terms>
)
But this doesn't work:
(|
(dn=cn=ppolicy,dc=capmon,dc=lan)
(dn=cn=Users,dc=capmon,dc=lan)
<more ORed terms>
)
even though the returned records look like they contain dn attributes. :-(
An OpenLDAP server's records don't have distinguishedName attributes, and neither of the filters above work against it.
Can I do something that will work against most major LDAP servers?
It's not possible to "Read" several entries in a single operation.
You can do a single search operation that will match and return several entries, but you cannot search on the "DN" itself.
I've seen several applications that are trying to get several entries by using complex filters such as "(|(cn=foo)(cn=bar)(cn=Fred Flintstone))", but this may result in more entries, unless all CN values are unique. It's not really a good practice either, as there are limits in the number of elements you can have in the filter, and such requests are usually not optimized in term of I/O.
It will be faster to read each invidual entry, as LDAP servers are optimized for such operations. If you want to reduce the latency, you can issue multiple asynchronous search operations on the same connection.
I'm using the Java API to query for all job ids using the code below
Bigquery.Jobs.List list = bigquery.jobs().list(projectId);
list.setAllUsers(true);
but it doesn't list me job ids that were run by Client ID for web applications (ie. metric insights) I'm using private key authentication.
Using the command line tool 'bq ls -j' in turn giving me only the metric insight job ids but not the ones ran with the private key auth. Is there a get all method?
The reason I'm doing this is trying to get better visibility into what queries are eating up our data usage. We have multiple sources of queries: metric insights, in house automation, some done manually, etc.
As of version 2.0.10, the bq client has support for API authorization using service account credentials. You can specify using a specific service account with the following flags:
bq --service_account your_service_account_here#developer.gserviceaccount.com \
--service_account_credential_store my_credential_file \
--service_account_private_key_file mykey.p12 <your_commands, etc>
Type bq --help for more information.
My hunch is that listing jobs for all users is broken, and nobody has mentioned it since there is usually a workaround. I'm currently investigating.
Jordan -- It sounds like you're honing in on what we want to do. For all access that we've allowed into our project/dataset we want to produce an aggregate/report of the "totalBytesProcessed" for all queries executed.
The problem we're struggling with is that we have a handful of distinct java programs accessing our data, a 3rd party service (metric insights) and 7-8 individual users who have query access via the web interface. Fortunately the incoming data only has one source so explaining the cost for that is simple. For queries though I am kinda blind at the moment (and it appears queries will be the bulk of the monthly bill).
It would be ideal if I can get the underyling data for this report with just one listing made with some single top level auth. With that I think from the timestamps and the actual SQL text I can attribute each query to a source.
One thing that might make this problem far easier is if there were more information in the job record (or some text adornment in the job_id for queries). I don't see that I can assign my own jobIDs on queries (perhaps I missed it?) and perhaps recording some source information in the job record would be possible? Just thinking out loud now...
There are three tables you can query for this.
region-**.INFORMATION_SCHEMA.JOBS_BY_{USER, PROJECT, ORGANIZATION}
Where ** should be replaced by your region.
Example query for JOBS_BY_USER in the eu region:
select
count(*) as num_queries,
date(creation_time) as date,
sum(total_bytes_processed) as total_bytes_processed,
sum(total_slot_ms) as total_slot_ms_cost
from
`region-eu.INFORMATION_SCHEMA.JOBS_BY_USER` as jobs_by_user,
jobs_by_user.referenced_tables
group by
2
order by 2 desc, total_bytes_processed desc;
Documentation is available at:
https://cloud.google.com/bigquery/docs/information-schema-jobs
I was writing a small test case to see what's more efficient, multiple small queries or a single big query, when I encountered this limitation.
The query looks like this:
(| (clientid=1) (clientid=2) (clientid=3) ...)
When the number of clients goes beyond 2103 ?! the LDAP server throws an error:
error code 1 - Operations Error
As far as I can tell the actual filter string length does not matter ~69KB (at least for Microsoft AD the length limit is 10MB). I tried with longer attribute names and got the same strange limit: 2103 operands
Does anyone have more information about this limitation?
Is this something specified in the LDAP protocol specification or is it implementation specific?
Is it configurable?
I tested this against IBM Tivoli Directory Server V6.2 using both the UnboundID and JNDI Java libraries.
It cannot be more than 8099 characters. See http://www-01.ibm.com/support/docview.wss?uid=swg21295980
Also, what you are doing is not a good practice. If there are common attributes these entries share (e.g., country code, department number, location, etc.), try to retrieve the results using common criteria given you by those attributes. If not, divide your search filter into smaller ones each of which is with few predicates and execute multiple searches. It depends the programming language you're using to do this, but try to execute each search in a separate thread to speed up your data retrieval process.
I've been using givenName+" "+surname for the CN field and I woke up screaming last night 'what about John Smith'? I can imagine any large organization employing multiple people with the same name. So of course this isn't going to work. What do people use instead?
EDIT Note: in inetOrgPerson the CN is part of the DN.
EDIT Note: in this situation I am expecting to grow to hundreds of thousands of user entries.
In a LDAP Directory, whatever if it's OpenLDAP or Active-Directory, a rule is that a DistinguishName (DN) must be unique, independently of the attribute (or the attributes) used to constitute the Relative Distinguish Name (RDN).
How do people make sure that it's unique :
I would say that in a small business the guy who creates the entry in the directory guarantee that it's unique, first by knowledge, second by preliminary search. If a duplicate appears he finds some solutions like 'John E Smith'. Using this solution if the name changes (marriage, divorce etc.), the LDAP record has to "move" from one DN to another. It's better to avoid changing the DN of an entry whenever possible, but in a small directory it's not important.
In a medium business the uniqueness is most of the time given by the employee ID coming from human resources. For example FR12345678. I saw, in big companies, people logging in with their employee ID. For the thing I describe here, it's more standart to use the uid attribute to name an object in spite of cn (but some directories don't let you choise of the naming attribute, I think it's a
X500 feature).
In most directories (not in AD) you can use more than one attribute to compose the RDN. For example sn=Assin+TelephoneNumber=1234 is a valid RDN in an openLDAP and it can make sense in a PBX.
One more thing
In some directories (designed for system administration) some attributes are tested by the server side as unique all over the tree. That's the case of sAMAccountName or userPrincipalName in Active-Directory and they are used for loging purpose. Using the CN attribute with "given-Name Name" oblige the administrators to guarantee uniqueness. You can use unique attribute in OpenLDAP for that in the database definition in slapd.conf, add :
# index since the unique overlay will search for matching mail attributes
index mail eq
overlay unique
unique_attributes mail
If unique overlay is not compiled in, you'll need to recompile with :
./configure ... --enable-unique
Adding to JPBlanc's answer with some of my experience. We have several ldap servers/trees where I work. Our AD server is using the DisplayName as the value of the CN. Out of 4K+ users we have only had a few instances where duplicates have occurred. I believe the default action there is to tack a 1 on the value if there is a dupe. It is surprisingly rare even with a high turn over rate in the largest section of that user base. We have two different e-directory trees that are linked to each other and those use the username. Username is first initial + last name. Any duplicates there have an incrementing number attached to them. As you can imagine that happens a lot with the Browns and the Smiths and other common names. Another tree that is an ADLDS (formerly ADAM) directory uses a uniquely generated number for each new entry as the CN. It is basically an auto-incremented number that is controlled by an external loading process. Lastly we have a directory for external partners (think independent agents) that uses a combination of email address + an id number as the CN.
I do a lot of maintenance work on the user bases and my least favorite scheme is the externally generated number. If I get a support call about Joe Brown in all of the other systems I can at least have an idea of where I need to browse to find him. Sure a simple search filter will give me all of the Browns but I still have to write it and execute it. So my advice is to use some part of the name for the CN and ensure uniqueness somehow. From an administration point of view it will be a bit easier. Really the CN is important but you'll be dealing with the rest of the user attributes far more so don't sweat it too badly.