I am trying to deal with being able to read a file hosted in S3 using a Java application. I have added the AmazonS3Client but it fails with when it goes to ProfileCredentialsProvider. I have my AWS credentials at C:/User/.aws (default), credentials are found but the following error is thrown: "Invalid property format: no '=' character". My credentiasl (download from Amazon) is a csv file separated by commas. Please can anybody tell me if I have to modify this file? Thanks in advance
Yes, you will need to change the format of that file; the ProfileCredentialsProvider class does not expect the CSV format, but instead expects the file to be formatted like one shown in the ProfilesConfigFile docs:
[default]
aws_access_key_id=testAccessKey
aws_secret_access_key=testSecretKey
Hope this helps!
If anybody ever stumbles upon this issue and cannot find any solutions, check if you have inline comments like [some-profile] #inline comment in your credentials file. Seems like it is not supported and breaks everything.
Related
I would like to know is there anyway we can encrypt the server.ssl.key-store-password value and store it in application.properties file instead of storing it in plain text.
i couldn't find any documentation on this. Any help on this is highly appreciated.
Thanks in advance.
Spring allows you to encrypt the properties file but the key for that encryption needs to be kept somewhere. This answer suggest keeping them in environment variables and points to a guide about how to encrypt them if you still want to.
You can use "jasypt-spring-boot-starter" for your need. All you to need to do are the following steps.
Download the "jasypt-spring-boot-starter" from maven central repo.
com.github.ulisesbocchio
jasypt-spring-boot-starter
x.x.x
In your Spring Boot start file where the "#SpringBootApplication" annotation is located, just include "#EnableEncryptableProperties". A point to note here is that once you place encryptable properties annotation on the main start file, all the property files of your application will be loaded and scanned by Jaspyt module for any property value that is marked starting with "ENC".
In your "application.properties" file there are few more configurations that needed to be added like below (all these are defaults and you can change these according to your requirement):
jasypt.encryptor.password=<Some password for encryption>
jasypt.encryptor.algorithm=PBEWITHHMACSHA256ANDAES_128
jasypt.encryptor.key-obtention-iterations=1000
jasypt.encryptor.pool-size=1
jasypt.encryptor.salt-generator-classname=org.jasypt.salt.RandomSaltGenerator
jasypt.encryptor.iv-generator-classname=org.jasypt.iv.RandomIvGenerator
jasypt.encryptor.string-output-type=base64
Once you are done with the above steps, now you can place your encrypted property value under the ENC(). Jasypt will scan values which are enclosed in ENC() and will try to decrypt the value.
For e.g.
spring.datasource.password=ENC(tHe0atcRsE+uOTxt2GmFYPXNHREch9R/12qD082gw7vv6bby5Rk)
I want to read an S3 file from my (local) machine, through Spark (pyspark, really). Now, I keep getting authentication errors like
java.lang.IllegalArgumentException: AWS Access Key ID and Secret
Access Key must be specified as the username or password
(respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId
or fs.s3n.awsSecretAccessKey properties (respectively).
I looked everywhere here and on the web, tried many things, but apparently S3 has been changing over the last year or months, and all methods failed but one:
pyspark.SparkContext().textFile("s3n://user:password#bucket/key")
(note the s3n [s3 did not work]). Now, I don't want to use a URL with the user and password because they can appear in logs, and I am also not sure how to get them from the ~/.aws/credentials file anyway.
So, how can I read locally from S3 through Spark (or, better, pyspark) using the AWS credentials from the now standard ~/.aws/credentials file (ideally, without copying the credentials there to yet another configuration file)?
PS: I tried os.environ["AWS_ACCESS_KEY_ID"] = … and os.environ["AWS_SECRET_ACCESS_KEY"] = …, it did not work.
PPS: I am not sure where to "set the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties" (Google did not come up with anything). However, I did try many ways of setting these: SparkContext.setSystemProperty(), sc.setLocalProperty(), and conf = SparkConf(); conf.set(…); conf.set(…); sc = SparkContext(conf=conf). Nothing worked.
Yes, you have to use s3n instead of s3. s3 is some weird abuse of S3 the benefits of which are unclear to me.
You can pass the credentials to the sc.hadoopFile or sc.newAPIHadoopFile calls:
rdd = sc.hadoopFile('s3n://my_bucket/my_file', conf = {
'fs.s3n.awsAccessKeyId': '...',
'fs.s3n.awsSecretAccessKey': '...',
})
The problem was actually a bug in the Amazon's boto Python module. The problem was related to the fact that MacPort's version is actually old: installing boto through pip solved the problem: ~/.aws/credentials was correctly read.
Now that I have more experience, I would say that in general (as of the end of 2015) Amazon Web Services tools and Spark/PySpark have a patchy documentation and can have some serious bugs that are very easy to run into. For the first problem, I would recommend to first update the aws command line interface, boto and Spark every time something strange happens: this has "magically" solved a few issues already for me.
Here is a solution on how to read the credentials from ~/.aws/credentials. It makes use of the fact that the credentials file is an INI file which can be parsed with Python's configparser.
import os
import configparser
config = configparser.ConfigParser()
config.read(os.path.expanduser("~/.aws/credentials"))
aws_profile = 'default' # your AWS profile to use
access_id = config.get(aws_profile, "aws_access_key_id")
access_key = config.get(aws_profile, "aws_secret_access_key")
See also my gist at https://gist.github.com/asmaier/5768c7cda3620901440a62248614bbd0 .
Environment variables setup could help.
Here in Spark FAQ under the question "How can I access data in S3?" they suggest to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
I cannot say much about the java objects you have to give to the hadoopFile function, only that this function already seems depricated for some "newAPIHadoopFile". The documentation on this is quite sketchy and I feel like you need to know Scala/Java to really get to the bottom of what everything means.
In the mean time, I figured out how to actually get some s3 data into pyspark and I thought I would share my findings.
This documentation: Spark API documentation says that it uses a dict that gets converted into a java configuration (XML). I found the configuration for java, this should probably reflect the values you should put into the dict: How to access S3/S3n from local hadoop installation
bucket = "mycompany-mydata-bucket"
prefix = "2015/04/04/mybiglogfile.log.gz"
filename = "s3n://{}/{}".format(bucket, prefix)
config_dict = {"fs.s3n.awsAccessKeyId":"FOOBAR",
"fs.s3n.awsSecretAccessKey":"BARFOO"}
rdd = sc.hadoopFile(filename,
'org.apache.hadoop.mapred.TextInputFormat',
'org.apache.hadoop.io.Text',
'org.apache.hadoop.io.LongWritable',
conf=config_dict)
This code snippet loads the file from the bucket and prefix (file path in the bucket) specified on the first two lines.
I have a file lets say test.txt, I encrypt it into mypgpfile.pgp. Now when I decrypt this using apache camel, it writes the output to mypgpfile.pgp. Is there a way in Camel to know the embeddedfilename? I want the output to be written to test.txt instead of mypfpfile.pgp.
The code sample is simple as follows.
from("direct:unencrypt")
.unmarshal().pgp(keyFileNameSec, keyUserid, keyPassword)
Is it possible to get the original file name by writing custom code using bouncycastle? Can someone please help with suggestions, since retaining the original file name is an important requirement for us. Thanks for your help in advance!
I am curious about the proper way to stop a user from activating my plugin if their system does not meet certain requirements. Doing the checks is easy and I don't need any help with that, I am more curious how to tell WordPress to exit and display an error message.
Currently I have tried both exit($error_message) and die($error_message) in the activation hook method. While my message is displayed and the plugin is not activated, a message saying Fatal Error is also displayed (see image below).
Does anyone know of a better way, that would display my message in a proper error box without displaying Fatal error, it just looks really bad for new users to see that.
Thanks for any help in advance.
This is a little undocumented, as you might have noticed. Instead of die(), do it like this:
$plugin = dirname(__FILE__) . '/functions.php';
deactivate_plugins($plugin);
wp_die('<p>The <strong>X</strong> plugin requires version WordPress 2.8 or greater.</p>','Plugin Activation Error',array('response'=>200,'back_link'=>TRUE));
The lines above wp_die() are to deactivate this plugin. Note that we use functions.php in this case because that's where I have my Plugin Name meta data comment declaration -- and if you use a different file, then change the code above. Note that the path is very specific for a match. So, if you want to see what your path would normally be, use print_r(get_option('active_plugins'));die(); to dump that out so that you know what path you need. Since I had a plugin_code.php where the rest of my plugin code was, and since it was in the same directory as functions.php, I merely had to do dirname(__FILE__) for the proper path.
Note that the end of the wp_die() statement is important because it provides a backlink and prevents an error 500 (which is the default Apache code for wp_die()).
It is only a idea though. Try checking the wordpress version and compare then use php to through custom exception/error. PHP 5.0 try catch can be a good way to do it. Here is some resources.
http://www.w3schools.com/php/php_exception.asp
http://php.net/manual/en/internals2.opcodes.throw.php
You can try the first link. It is pretty basic. Thanks! hope the information will be helpful.
I'm analyzing my code (C#, desktop application) with CAT.NET Code Analysis and getting "Sanitize the file path prior to passing it to file system routines" message when dealing with file names.
What I don't understand is that to ensure the file name is valid, I use:
void SomeMethod(String filename)
{
filename = System.IO.Path.GetFullPath(filename);
// ... Do stuff
}
Isn't it a "magic solution" to solve problems with invalid file names ? I've read something similar here (first answer), but in my case I'm dealing only with local files, well, something very basic, so...
So why I'm getting this message and how to do to avoid getting it?
I know this is an old question, but I've just come across something that may be helpful specifically related to the CAT.Net error message.
In a blog post about the CAT.Net Data Flow Rules, they have this to say about the FileCanonicalizationRule:
Description
User input used in the file handling routines can potentially lead to
File Canonicalization vulnerability. Code is particularly susceptible
to canonicalization issues if it makes any decisions based on the name
of a resource that is passed to the program as input. Files, paths,
and URLs are resource types that are vulnerable to canonicalization
because in each case there are many different ways to represent the
same name.
Resolution
Sanitize the file path prior to passing it to file handling routines.
Use Path.GetInvalidFileNameChars or Path.GetInvalidPathChars to get
the invalid characters and remove them from the input. More
information can be found at
http://msdn.microsoft.com/en-us/library/system.io.path.getinvalidfilenamechars.aspx.
So, they suggest that you use Path.GetInvalidFileNameChars and Path.GetInvalidPathChars to validate your paths.
Note that their suggestion is to remove the invalid characters. While this will indeed make the path valid, it may cause unexpected behaviour for the user. As the comments on this question/answer suggest it's probably better to quit early and tell the user that their path is invalid, rather than doing something unexpected with their input (like removing bad characters and using the modified version).
If the filename comes from a user, it could be something like "../../../../etc/passwd" - the error message is telling you that you need to sanitize it so that it can't get to directories it's not supposed to.