I am trying to execute 3 to 4 roles in ansible, parllel all at a time, but I want only one specific role to have forks option set to 1 whereas the other roles can have any number of forks
I'm not sure what your exact end goal is but I suspect the serial directive is what you're looking for to limit how the execution operates. Keep in mind you may have to split into multiple plays within the playbook depending on how you have things setup now.
http://docs.ansible.com/ansible/latest/playbooks_delegation.html#rolling-update-batch-size
Related
I'm trying to use HAProxy for rate-limiting.
I need to keep track of several endpoints and limit them individually.
So far I was using general purpose counters. However, there is only 3 of them, sc0 to sc2.
In the documentation, it mentions that all the operations on these counters take an optional table parameter, but it's not clear, then, if I can track different things on different tables, but using the same counters.
In other words: is the limit of 3 general purpose counters global, or per sticky table?
If, after proper table definition and track instructions, I do
sc1_inc_gpc0(table1)
(and, under different conditions)
sc1_inc_gpc0(table2)
And then have 2 acl rules like
acl X sc1_get_gpc0(table1) gt 1
acl Y sc1_get_gpc0(table2) gt 1
Will the two acls work indepentently, or they would to all effects track the same counter?
Thanks for all help!
(In case you are wondering: for a number of reasons, at the moment I could not use a different solution than HAProxy for rate-limiting)
Self answered after looking at the source code and testing.
Yes it is possible to use the same counter on different tables
Moreover, you can also increment the number of available counters at build time. The default is 3, but it can be set up to 10 for sure. Then generic versions of the functions like sc_gpc0_rate(<ctr>[,<table>]) can be used, passing the index of the new counter as first argument.
In my application I store users as user:n where n is a unique ID.
When a new user is created I increment a global variable such as user_count and use that ID as user:n.
But, I have an issue where I need to ensure an email is not already in use. I've done some reading around and the only way I can see how to do this is to:
1) Loop through the users. But, I am not keen on this solution as it could cause slower performance right?
2) Create a lookup that contains a list of email addresses used.
Both solutions seem a bit strange to me as I come from an SQL background.
Are these the only options available? I also have to do the same check for usernames too.
You could use Sets:
On registration: sadd taken_emails "john#example.com"
And testing with: sismember taken_emails "bob#exmaple.com"
Note that you have a possible race-condition where two users try to use the same email at the same time, both test and get "free" and then both register with it. You could use a lock to make sure they don't both get it, or make the registration operation atomic with either WATCH/MULTI/EXEC or with a lua script.
If I have a row of data like:
1, 2, 3
I can create a checksum value that is the sum of all of the columns, 1 + 2 + 3 = 6. We can store this value with the row in the 4th column:
1, 2, 3, 6
I can then write a program to check to see if any of the values in the columns changed accidentally if the sum of the columns don't match the checksum value.
Now, I'd like to take this a step further. Let's say I have a table of values that anyone has read/write access to where the last column of data is the sum of the previous columns as described earlier.
1, 2, 3, 6
Let's say someone wants to be sneaky and change the value in the third column
1, 2, 9, 6
The checksum is easy to reproduce so the sneaky individual can just change the checksum value to 1 + 2 + 9 = 12 so that this row appears not to be tampered with.
1, 2, 9, 12
Now my question is, how can I make a more sophisticated checksum value so that a sneaky individual can't make this type of change without making the checksum no longer valid? Perhaps I could create a blackbox exe that given the first three values of the row can give a checksum that is a little more sophisticated like:
a^2 + b^2 + c^2
But while this logic is unknown to a sneaky user, he/she could still input the values into the black box exe and get a valid checksum back.
Any ideas on how I can make sure all rows in a table are untampered with? The method I'm trying to avoid is saving a copy of the table every time it is modified legitimately using the program I am creating. This is possible, but seems like a very unelegant solution. There has to be a better way, right?
Using basic math your checksum is invalid:
a^2 +b^2 +c^2
a=0,b=0,c=2 = checksum 4
a=2,b=0,c=0 = checksum 4
If you want a set of "read-only" data to the users, consider using materialized views. A materialized view will compute the calculation a head of time i.e. your valid data and serve that to the users, while your program can do modifications in the background.
Further this is the reason why privileges exist, if you only supply accounts that cannot modify the database for instance read-only access, this mitigates the issue of someone tampering with data. Also you cannot fully prevent a malicious user from tampering with data only make them jump through several hoops in hopes they get bored / blocked temporarily.
There is no silver bullet for security, what you can do is use a defense in depth mindset that would consist of the following features:
Extensive Logging,
Demarcation of responsibilities,
Job rotation,
Patch management,
Auditing of logs (goes together with logging, but someone actually has to read them),
Implement a HIPS system (host intrusion prevention system),
Deny outside connections to the database
The list can go on quite extensively.
You seem to be asking, "how can I give a program a different set of security permissions to the user running it?" The way to do this is to make sure the program is running in a different security context to the user. Ways of doing this vary by platform.
If you have multiple machines, then running a client server architecture can help. You expose a controlled API through the server, and it has the security credentials for the database. Then your user can't make arbitrary requests.
If you're the administrator of the client machine, and the user isn't then you may be able to have separate processes doing something similar. E.g. a daemon in unix. I think DCOM in windows lets you do something like this.
Another approach is to expose your API through stored procedures, and only grant access to these, rather than direct access to the table.
Having controlled access to a limited API may not be enough. Consider, for example, a table that stores High Scores in a game. It doesn't matter that it can only be accessed through a ClaimHighScore API, if the user can enter arbitrary values. The solution for this in games is usually complicated. The only approach I've heard of that works is to define the API in terms of a seed value that gave the initial game state, and then a set of inputs with timestamps. The server then has to essentially simulate the game to verify the score.
Users should not have unconstrained write access to tables. Better would be to create sprocs for common CRUD operations. This would let you control which fields they can modify, and if you insist you could update a CRC() checksum or other validation.
This would be a big project, so it may not be practical right now - but it's how things should be done.
Although your question is based on malicious entries to a database the use of the MOD11 can find inaccurate or misplaced values.
The following MySQL statement and SQLfiddle illustrate this
SELECT id, col1, col2, col3, col4, checknum,
9 - MOD(((col1*5)+(col2*4)+(col3*3)+(col4*2) ),9)
AS Test FROM `modtest` HAVING checknum =Test
I'm using the Java API to query for all job ids using the code below
Bigquery.Jobs.List list = bigquery.jobs().list(projectId);
list.setAllUsers(true);
but it doesn't list me job ids that were run by Client ID for web applications (ie. metric insights) I'm using private key authentication.
Using the command line tool 'bq ls -j' in turn giving me only the metric insight job ids but not the ones ran with the private key auth. Is there a get all method?
The reason I'm doing this is trying to get better visibility into what queries are eating up our data usage. We have multiple sources of queries: metric insights, in house automation, some done manually, etc.
As of version 2.0.10, the bq client has support for API authorization using service account credentials. You can specify using a specific service account with the following flags:
bq --service_account your_service_account_here#developer.gserviceaccount.com \
--service_account_credential_store my_credential_file \
--service_account_private_key_file mykey.p12 <your_commands, etc>
Type bq --help for more information.
My hunch is that listing jobs for all users is broken, and nobody has mentioned it since there is usually a workaround. I'm currently investigating.
Jordan -- It sounds like you're honing in on what we want to do. For all access that we've allowed into our project/dataset we want to produce an aggregate/report of the "totalBytesProcessed" for all queries executed.
The problem we're struggling with is that we have a handful of distinct java programs accessing our data, a 3rd party service (metric insights) and 7-8 individual users who have query access via the web interface. Fortunately the incoming data only has one source so explaining the cost for that is simple. For queries though I am kinda blind at the moment (and it appears queries will be the bulk of the monthly bill).
It would be ideal if I can get the underyling data for this report with just one listing made with some single top level auth. With that I think from the timestamps and the actual SQL text I can attribute each query to a source.
One thing that might make this problem far easier is if there were more information in the job record (or some text adornment in the job_id for queries). I don't see that I can assign my own jobIDs on queries (perhaps I missed it?) and perhaps recording some source information in the job record would be possible? Just thinking out loud now...
There are three tables you can query for this.
region-**.INFORMATION_SCHEMA.JOBS_BY_{USER, PROJECT, ORGANIZATION}
Where ** should be replaced by your region.
Example query for JOBS_BY_USER in the eu region:
select
count(*) as num_queries,
date(creation_time) as date,
sum(total_bytes_processed) as total_bytes_processed,
sum(total_slot_ms) as total_slot_ms_cost
from
`region-eu.INFORMATION_SCHEMA.JOBS_BY_USER` as jobs_by_user,
jobs_by_user.referenced_tables
group by
2
order by 2 desc, total_bytes_processed desc;
Documentation is available at:
https://cloud.google.com/bigquery/docs/information-schema-jobs
Let say I have a community with 50 000 members.
So there are one Redis set called community_###_members with 50 000 sha1 keys of users and for each user his own set user_###_communities exists with sha1 hash of community above
In some time I decide to delete community.. What is the best algorithm to kill all members?
Thanks.
assuming you really want to actually delete members of a community that does not exist, it's something like (python code example):
memebrs = redis.smembers('community_members')
pipe = redis.pipeline()
pipe.delete('community_memebrs')
for member in members:
pipe.delete(member)
pipe.execute()
if the community is very large you might want to do that N memebrs at a time, so the server will not stall until you finish all.