Error -
*** Failed to verify remote log exists s3://airflow_test/airflow-logs/demo/task1/2022-05-13T18:20:45.561269+00:00/1.log.
An error occurred (403) when calling the HeadObject operation: Forbidden
Dockerfile -
FROM apache/airflow:2.2.3
COPY /airflow/requirements.txt /requirements.txt
RUN pip install --no-cache-dir -r /requirements.txt
RUN pip install apache-airflow[crypto,postgres,ssh,s3,log]
USER root
# Update aptitude with new repo
RUN apt-get update
# Install software
RUN apt-get install -y git
USER airflow
Under connection UI -
Connection Id * - aws_s3_log_storage
Connection Type * - S3
Host - <My company's internal link>. (ex - https://abcd.company.com)
Extra - {"aws_access_key_id": "key", "aws_secret_access_key": "key", "region_name": "us-east-1"}
Under values.yaml -
config:
logging:
remote_logging: 'True'
remote_base_log_folder: 's3://airflow_test/airflow-logs'
remote_log_conn_id: 'aws_s3_log_storage'
logging_level: 'INFO'
fab_logging_level: 'WARN'
encrypt_s3_logs: 'False'
host: '<My company's internal link>. (ex - https://abcd.company.com)'
colored_console_log: 'False'
How did I created the bucket?
Installed awscli
used the commands -
1. aws configure
AWS Access Key ID: <access key>
AWS Secret Access Key: <secret key>
Default region name: us-east-1
Default output format:
2. aws s3 mb s3://airflow_test --endpoint-url=<My company's internal link>. (ex - https://abcd.company.com)
I am not getting a clue on how to resolve the error. I am actually very new to airflow and helm charts.
I had same error message with you. your account or Key might not have enough permission for accessing S3 bucket.
Please check your role has enough permissions below.
"s3:PutObject*",
"s3:PutObjectAcl",
"s3:PutObjectVersionAcl",
"s3:GetObject*",
"s3:ListObject*",
"s3:ListBucket*",
"s3:PutBucket*",
"s3:GetBucket*",
"s3:DeleteObject
I'm trying to set up Microsoft AD like user repository for IBM MQ v9 Queue Manager , but without success. I read the document https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.ref.adm.doc/q085490_.htm, but it's very unclear with all those diagrams, dashes and arrows. My final goal is to have ability to grant or rewoke authorizations based od AD groups. Can someone give me complete commands example how to configure queue manager to use AD for user repository?
IBM MQ is v9.0.0.0 and runs on CentOS v7. Active Directory is on Windows Server 2019 machine.
I tried to set AUTHINFO with MQSC commands. All commands are executed without problems. After that I refreshed security and tried to grant authorizations with setmqaut command, but unsuccessful.
I tried with this below MQSC commands:
DEFINE AUTHINFO(MY.AD.CONFIGURATION) AUTHTYPE(IDPWLDAP) AUTHORMD(SEARCHGRP) FINDGRP(member) CONNAME('192.168.100.100') BASEDNG('OU=Groups,OU=MyCompany,DC=mycompany,DC=us') SHORTUSR('sAMAccountName') LDAPUSER('mybinduser') LDAPPWD('mypassword')
ALTER QMGR CONNAUTH(MY.AD.CONFIGURATION)
REFRESH SECURITY TYPE(CONNAUTH)
setmqaut -m MY.QUEUE.MANAGER -t qmgr -g myadgroup +all
After I execute command:
setmqaut -m MY.QUEUE.MANAGER -t qmgr -g myadgroup +all
This error is displyed i console: AMQ7026: A principal or group name was invalid.
And these below lines are recorded in queue manager log:
AMQ5531: Error locating user or group in LDAP
EXPLANATION:
The LDAP authentication and authorization service has failed in the ldap_search
call while trying to find user or group 'myadgroup '. Returned count is 0.
Additional context is 'rc = 87 (Bad search filter)
[(&(objectClass=groupOfNames)(=myadgroup ))]'.
ACTION:
Specify the correct name, or fix the directory configuration. There may be
additional information in the LDAP server error logs.
----- amqzfula.c : 2489 -------------------------------------------------------
On Active Directory side these lines are recorded in log:
An account failed to log on.
Subject:
Security ID: SYSTEM
Account Name: MYADSERVER$
Account Domain: MYDOMAINNAME
Logon ID: 0x3E7
Logon Type: 3
Account For Which Logon Failed:
Security ID: NULL SID
Account Name: mybinduser
Account Domain: MYDOMAINNAME
Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xC000006D
Sub Status: 0xC000006A
Process Information:
Caller Process ID: 0x280
Caller Process Name: C:\Windows\System32\lsass.exe
Network Information:
Workstation Name: MYADSERVER
Source Network Address: 192.168.100.101
Source Port: 55592
Detailed Authentication Information:
Logon Process: Advapi
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Transited Services: -
Package Name (NTLM only): -
Key Length: 0
Here beleow is output of the command DIS AUTHINFO(MY.AD.CONFIGURATION) ALL
AMQ8566: Display authentication information details.
AUTHINFO(MY.AD.CONFIGURATION) AUTHTYPE(IDPWLDAP)
ADOPTCTX(NO) DESCR( )
CONNAME(192.168.100.100) CHCKCLNT(REQUIRED)
CHCKLOCL(OPTIONAL) CLASSGRP( )
CLASSUSR( ) FAILDLAY(1)
FINDGRP(MEMBER) BASEDNG(OU=Groups,OU=MyCompany,DC=mycompany,DC=us)
BASEDNU( )
LDAPUSER(CN=mybinduser,OU=System,OU=Users,OU=MyCompany,DC=mycompany,DC=us)
LDAPPWD( ) SHORTUSR(sAMAccountName)
GRPFIELD( ) USRFIELD( )
AUTHORMD(SEARCHGRP) NESTGRP(NO)
SECCOMM(NO) ALTDATE(2019-07-25)
ALTTIME(08.14.20)
Here below is output from LdapAuthentication.jar tool:
java -jar LdapAuthentication.jar ldap://192.168.100.100:389 CN=mybinduser,OU=System,OU=Users,OU=MyCompany,DC=mycompany,DC=us mybinduserpassword OU=MyCompany,DC=mycompany,DC=us sAMAccountName adminusername adminpassword
#WMBL3: successful bind
#WMBL3: successfull search Starting Authentication Found the user, DN is CN=adminusername,OU=MyCompany,OU=Users,OU=MyCompany,DC=mycompany,DC=us
#WMBL3 : check if the password is correct
#WMBL3: successful authentication
#WMBL3 : Commands for WebUI ldap authentication :
1. mqsisetdbparms <INodeName> -n ldap::LDAP -u "CN=mybinduser,OU=System,OU=Users,OU=MyCompany,DC=mycompany,DC=us" -p mybinduserpassword
Or
mqsisetdbparms <INodeName> -n ldap::192.168.100.100 -u "CN=mybinduser,OU=System,OU=Users,OU=MyCompany,DC=mycompany,DC=us" -p mybinduserpassword
2. mqsichangeproperties <INodeName> -b webadmin -o server -n ldapAuthenticationUri -v \"ldap://192.168.100.100:389/OU=MyCompany,DC=mycompany,DC=us?sAMAccountName\"
3. mqsiwebuseradmin <INodeName> -c -u adminusername -x -r <sysrole for eg: local userid >
Here below is qmanager log after I applied changes in my AUTHINFO what you suggested Jul 25.
AMQ5531: Error locating user or group in LDAP
EXPLANATION:
The LDAP authentication and authorization service has failed in the ldap_search
call while trying to find user or group 'wasadmin'. Returned count is 0.
Additional context is 'rc = 1 (Operations error)
[(&(objectClass=GROUP)(SAMACCOUNTNAME=wasadmin))]'.
ACTION: Specify the correct name, or fix the directory configuration. There may be
additional information in the LDAP server error logs.
This is myadgroup full DN:
CN=myadgroup,OU=System,OU=Groups,OU=MyCompany,DC=mycompany,DC=us
This is output of the setmqaut command with full group DN:
setmqaut -m MY.QUEUE.MANAGER -t qmgr -g 'CN=myadgroup,OU=System,OU=Groups,OU=MyCompany,DC=mycompany,DC=us' +all
AMQ7047: An unexpected error was encountered by a command. Reason code is 2063.
And this is qmanager log after that command was executed:
AMQ5531: Error locating user or group in LDAP
EXPLANATION: The LDAP authentication and authorization service has failed in the ldap_search call while trying to find user or group 'CN=myadgroup,OU=System,OU=Groups,OU=MyCompany,DC=mycompany,DC=us'.
Returned count is 0.
Additional context is 'rc = 1 (Operations error) [(objectClass=groupOfNames)]'.
ACTION:
Specify the correct name, or fix the directory configuration. There may be
additional information in the LDAP server error logs.
If I try with CLASSGRP(GROUP) output of the setmqaut is:
AMQ7047: An unexpected error was encountered by a command. Reason code is 2063.
And qmqnager log is:
AMQ5531: Error locating user or group in LDAP
EXPLANATION: The LDAP authentication and authorization service has failed in the
ldap_search call while trying to find user or group
'CN=myadgroup,OU=System,OU=Groups,OU=MyCompany,DC=mycompany,DC=us'.
Returned count is 0.
Additional context is 'rc = 1 (Operations error) [(objectClass=GROUP)]'.
ACTION:
Specify the correct name, or fix the directory configuration. There may be
additional information in the LDAP server error logs.
Below is my last configured authinfo object:
AMQ8566: Display authentication information details.
AUTHINFO(MY.AD.CONFIGURATION) AUTHTYPE(IDPWLDAP)
ADOPTCTX(YES) DESCR( )
CONNAME(192.168.100.100) CHCKCLNT(OPTIONAL)
CHCKLOCL(OPTIONAL) CLASSGRP(group)
CLASSUSR(USER) FAILDLAY(1)
FINDGRP(member)
BASEDNG(OU=Groups,OU=MyCompany,DC=mycompany,DC=us)
BASEDNU(OU=Users,OU=MyCompany,DC=mycompany,DC=us)
LDAPUSER(CN=mybinduser,OU=System,OU=Users,OU=MyCompany,DC=mycompany,DC=us)
LDAPPWD( ) SHORTUSR(sAMAccountName)
GRPFIELD(sAMAccountName) USRFIELD(sAMAccountName)
AUTHORMD(SEARCHGRP) NESTGRP(NO)
SECCOMM(NO) ALTDATE(2019-08-07)
ALTTIME(08.44.40)
Based on the your output I noted that you did not set LDAPPWD which is used by MQ to authenticate the LDAPUSER that you specified.
This is supported by the windows error you provided:
Account For Which Logon Failed:
Security ID: NULL SID
Account Name: mybinduser
Account Domain: MYDOMAINNAME
Failure Information:
Failure Reason: Unknown user name or bad password.
In the output of LdapAuthentication.jar it appears that you have the correct password available:
CN=mybinduser,OU=System,OU=Users,OU=MyCompany,DC=mycompany,DC=us mybinduserpassword
You can either specify the LDAPPWD or you can blank out your LDAPUSER and see if your AD allows anonymous bind (this is rare).
I noted that you have some other fields left blank that probably need to be filled in. I also suggest you always use ADOPTCTX(YES).
Below is my suggested updates to your AUTHINFO object:
ALTER AUTHINFO(MY.AD.CONFIGURATION) +
AUTHTYPE(IDPWLDAP) +
AUTHORMD(SEARCHGRP) +
FINDGRP('member') +
ADOPTCTX(YES) +
CONNAME(192.168.100.100) +
CHCKCLNT(REQUIRED) +
CHCKLOCL(OPTIONAL) +
CLASSGRP(GROUP) +
CLASSUSR(USER) +
FAILDLAY(1) +
BASEDNG('OU=MyCompany,DC=mycompany,DC=us') +
BASEDNU('OU=MyCompany,DC=mycompany,DC=us') +
LDAPUSER('CN=mybinduser,OU=System,OU=Users,OU=MyCompany,DC=mycompany,DC=us') +
LDAPPWD(mybinduserpassword) +
SHORTUSR(sAMAccountName) +
GRPFIELD(sAMAccountName) +
USRFIELD(sAMAccountName) +
NESTGRP(NO) +
SECCOMM(NO)
*Note I have not tested this against AD, but I have setup IIB to authenticate the WebUI/REST calls against AD and also took inspiration from two presentations/write ups from Mark Taylor from IBM:
MQ Integration with Directory Services - Presented at MQTC v2.0.1.6
MQdev Blog: IBM MQ - Using Active Directory for authorisation in Unix queue managers
Within a bash script, I am invoking yarn application command in order to get the current applications running on a Cloudera Hadoop cluster secured by Kerberos. In case my application is not running, it is necessary to restart it:
spark_rtp_app_array=( $(yarn application --list -appTypes SPARK -appStates ACCEPTED,RUNNING | awk -F "\t" ' /'my_user'/ && /'my_app'/ {print $1}') )
Whenever the Kerberos ticket has ended I need to invoke kinit command, in order to renew that ticket before calling yarn application --list:
kinit -kt my_keytab_file.keytab my_kerberos_user
Otherwise, I could end with an authentication error which keeps repeating in an undefinite way with the following traces:
19/02/13 15:00:22 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS in\
itiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
19/02/13 15:00:22 WARN security.UserGroupInformation: PriviledgedActionException as:my_kerberos_user (auth:KERBEROS) cause:java.io\
.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechan\
ism level: Failed to find any Kerberos tgt)]
[...]
Is there any way of setting a maximum number of connection retries to YARN?
The bash script is being executed in a cron task, so it should not be hung in any way.
I implemented a REST api in django with django-rest-framework,on localhost working fine with successful results.
When pushing this up to an existing AWS elastic beanstalk instance, I received:
{
"detail": "Authentication credentials were not provided."
}
For solution I followed this question : Authorization Credentials Stripped
But when I push mycode on aws EB I am getting this error :
Pipeline failed with error "Service:AmazonRDS, is not authorized to perform: rds:DescribeDBEngineVersions"
I tried lots of solutions but every time I am getting this error.
Note: I am using python3.6
I got the answer of my problem.
I set the RDS policy and create new custom_wsgi.config file on .ebextensions directory and write command :
files:
"/etc/httpd/conf.d/wsgihacks.conf":
mode: "000644"
owner: root
group: root
content: |
WSGIPassAuthorization On
I have registered two users in Keyrock (the global fiware labs instance at https://account.lab.fiware.org)
User Robin has an organisation Robin-Cloud
User Robin is owner of an application Babbler
The application Babbler has authorized users "Robin" and "Robin viewer" (see screenshot)
User "Robin viewer" is a member of the same organisation as user "Robin"
Just to be sure i authorized the whole organisation that "Robin viewer" belongs to.
Both users have the same roles.
I can authenticate user "Robin" using a shell script to get an Access token. In the shell script i pass in the Applications Client ID and Client secret. I also pass in the username and password of User "Robin". The shell script is here (altered copy of this https://raw.githubusercontent.com/Bitergia/fiware-chanchan-docker/master/images/pep-wilma/4.3.0/auth-token.sh ):
#!/bin/bash
if [ $# -lt 2 ] ; then
echo "auth-token: missing parameters."
echo "Usage: auth-token <user-email> <password>"
exit 1
fi
# Retrieve X-Auth-Token to make request against the protected resource
function get_token () {
if [ $# -lt 2 ] ; then
echo "get_token: missing parameters."
echo "Usage: get_token <user-email> <password>"
exit 1
fi
local _user=$1
local _pass=$2
# Retrieve Client ID and client Secret Automatically
CLIENT_ID="e2c095aa42414e75b9ac4d760f4c625a"
CLIENT_SECRET="****"
# Generate the Authentication Header for the request
AUTH_HEADER="$(echo -n ${CLIENT_ID}:${CLIENT_SECRET} | base64)"
# Define headers
CONTENT_TYPE="\"Content-Type: application/x-www-form-urlencoded\""
AUTH_BASIC="\"Authorization: Basic ${AUTH_HEADER}\""
# Define data to send
DATA="'grant_type=password&username=${_user}&password=${_pass}&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}'"
# Create the request
REQUEST="curl -s --insecure -i --header ${AUTH_BASIC} --header ${CONTENT_TYPE} -X POST https://account.lab.fiware.org/oauth2/token -d ${DATA}"
XAUTH_TOKEN="$(eval ${REQUEST})"
echo "Request: ${REQUEST}"
echo "X-Auth-Token for '${_user}': ${XAUTH_TOKEN}"
}
get_token $1 $2
However
I cannot get an access token for User "Robin viewer". The message i get from Keyrock is:
{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
I assumed authorizing user "Robin viewer" for the Babbler app in the Keyrock user interface would be enough. What am i missing here?
It seems the /oauth2/token?grant_type=password combination is only meant for resource owners, which is exactly what i am seeing.
So it seems we are limited to requesting an access token with this:
/oauth2/authorize?response_type=token&client_id=....&redirect_uri=....
Which forces us through the web login screen of the Horizon Front-End. We are starting the login flow from a smartphone so this is far from ideal because the Horizon screens are NOT responsive at all :(
[Edit]
The error seems to be due to special characters in the email address that were not encoded (we have a plus sign because gmail ignores anything after the plus, allowing us to create multiple email addresses and still receive them at a real email address: e.g. username+test#gmail.com arrives at username#gmail.com).
So our problem seems to have gone away (or maybe the latest Keyrock update on the global fiware instance solved it for us?). We can now use /oauth2/token?grant_type=password after all.