UserPrivilegedException, while creating a directory in HDFS as a specific user, through JAVA - apache

I a'm using CDH 4.7. I am trying to create a folder in HDFS in /user/cloudera. But the UserPrivilegedException is thrown. Below is my code (#copied)
public static void main(String args[]) {
try {
UserGroupInformation ugi =
UserGroupInformation.createProxyUser("cloudera", UserGroupInformation.getLoginUser());
System.out.println(ugi.getUserName());
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
System.out.println("aaa");
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://localhost.localdomain:8020/user/cloudera");
conf.set("hadoop.job.ugi", "cloudera");
FileSystem fs = FileSystem.get(conf);
fs.createNewFile(new Path("/user/cloudera/test"));
Path path = new Path("/user/cloudera/Hbasesyntax.txt");
FileStatus[] status = fs.listStatus(path);
for(int i=0;i<status.length;i++){
System.out.println(status[i].getPath());
}
return null;
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
Then, I even tried accessing the file (Hbasesyntax.txt) which I -put through the terminal. But, not able to get the file information from that too. Am I missing anything.
Below is the exception thrown:
Feb 23, 2015 1:18:27 AM org.apache.hadoop.security.UserGroupInformation doAs
SEVERE: **PriviledgedActionException as:cloudera via cloudera cause:java.io.IOException: Mkdirs failed to create /user/cloudera
java.io.IOException: Mkdirs failed to create /user/cloudera
at** org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:378)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:564)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:545)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:507)
at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:647)
at HDFSFileOperations.GetIntoFS$1.run(GetIntoFS.java:32)
at HDFSFileOperations.GetIntoFS$1.run(GetIntoFS.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1 190)
at HDFSFileOperations.GetIntoFS.main(GetIntoFS.java:21)

Related

how to jdbc to hive using java on ubuntu?

Ubuntu 16.04.1 LTS
hadoop 3.3.1
Hive 2.3.9
I have a java file:
public class HiveCreateDb {
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
public static void main(String[] args) throws SQLException {
// Register driver and create driver instance
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
System.exit(1);
}
// get connection
Connection con = DriverManager.getConnection("jdbc:hive://localhost:10000/default", "", "");
Statement stmt = con.createStatement();
stmt.executeQuery("CREATE DATABASE userdb");
System.out.println("Database userdb created successfully.");
con.close();
}
}
I put this java file on ubuntu folder,and then run
javac HiveCreateDb.java
HiveCreateDb.java:14: error: unreported exception ClassNotFoundException; must be caught or declared to be thrown
Class.forName(driverName);
^
1 error
I have downloaded hive-jdbc-3.1.2.jar,where shall I put this jar?
javac -classpath jars/hive-jdbc-2.3.9.jar source/HiveCreateDb.java

java.io.FileNotFoundException (Read-only file system) // Uploading files to S3

I am trying to create csv files from a list of maps and uploading them to S3 bucket through a lambda function. Following is the code:
public void createCSV(List<Map<String, AttributeValue>> changedRecords, Context context, String tableName)
throws IOException {
Calendar calendar = Calendar.getInstance();
SimpleDateFormat formatter = new SimpleDateFormat("yyyyMMddHHmmss");
String outputName = tableName + "_" + formatter.format(calendar.getTime()) + ".csv";
List<String> headers = changedRecords.stream().flatMap(map -> map.keySet().stream()).distinct()
.collect(Collectors.toList());
try (FileWriter writer = new FileWriter(outputName, true);) {
for (String string : headers) {
writer.write(string);
writer.write(",");
}
writer.write("\r\n");
for (Map<String, AttributeValue> lmap : changedRecords) {
for (Entry<String, AttributeValue> string2 : lmap.entrySet()) {
writer.write(string2.getValue().getS());
writer.write(",");
}
writer.write("\r\n");
}
}
catch (Exception e) {
e.printStackTrace();
}
s3.putObject(new PutObjectRequest("bucket_name", "data/" + outputName, outputName));
}
Getting the following fileNotFound exception:
java.io.FileNotFoundException: data_20200227192207.csv (Read-only file
system)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:133)
at java.io.FileWriter.(FileWriter.java:78)
at com.amazonaws.lambda.demo.PLMLambda.createCSV(PLMLambda.java:84)
at com.amazonaws.lambda.demo.PLMLambda.handleRequest(PLMLambda.java:54)
at com.amazonaws.lambda.demo.PLMLambda.handleRequest(PLMLambda.java:1)
at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:178)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:906)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:341)
at lambdainternal.AWSLambda.(AWSLambda.java:63)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:114)
Change the line:
try (FileWriter writer = new FileWriter(outputName, true);) {
to
try (FileWriter writer = new FileWriter("/tmp" + outputName, true);) {
In Lambda you can only write to the /tmp directory.
If your CSV content is not so huge (like 4GB), you can simple use StringWriter instead of FileWriter with AWS Lambda, and then just put directly the string to S3.

Unable to cleanup Infinispan DefaultCacheManager in state FAILED

I am getting this Exception when trying to restart CacheManager, that failed to start.
Caused by: org.infinispan.jmx.JmxDomainConflictException: ISPN000034: There's already a JMX MBean instance type=CacheManager,name="DefaultCacheManager" already registered under 'org.infinispan' JMX domain. If you want to allow multiple instances configured with same JMX domain enable 'allowDuplicateDomains' attribute in 'globalJmxStatistics' config element
at org.infinispan.jmx.JmxUtil.buildJmxDomain(JmxUtil.java:53)
I think it's a bug, but am I correct?
The version used is 9.0.0.Final.
EDIT
The error can be seen using this code snippet.
import org.infinispan.configuration.cache.*;
import org.infinispan.configuration.global.*;
import org.infinispan.manager.*;
class Main {
public static void main(String[] args) {
System.out.println("Starting");
GlobalConfigurationBuilder global = GlobalConfigurationBuilder.defaultClusteredBuilder();
global.transport()
.clusterName("discover-service-poc")
.initialClusterSize(3);
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.clustering().cacheMode(CacheMode.REPL_SYNC);
DefaultCacheManager cacheManager = new DefaultCacheManager(global.build(), builder.build(), false);
try {
System.out.println("Starting cacheManger first time.");
cacheManager.start();
} catch (Exception e) {
e.printStackTrace();
cacheManager.stop();
}
try {
System.out.println("Starting cacheManger second time.");
System.out.println("startAllowed: " + cacheManager.getStatus().startAllowed());
cacheManager.start();
System.out.println("Nothing happening because in failed state");
System.out.println("startAllowed: " + cacheManager.getStatus().startAllowed());
} catch (Exception e) {
e.printStackTrace();
cacheManager.stop();
}
cacheManager = new DefaultCacheManager(global.build(), builder.build(), false);
cacheManager.start();
}
}

Passing java.security.auth.login.config to Mobilefirst Patform Server

How can we pass following parameter to Mobilefirst Development Server?
-Djava.security.auth.login.config=login.config
I have tried adding it to jvm.options file, and it seems it is passed as parameter without effect.
Following is the code I am trying to execute, and sample of login.config file.
Java code to execute in login module or adapter.
LoginContext context = new LoginContext("SampleClient", new CallbackHandler() {
#Override
public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
NameCallback callBack = (NameCallback) callbacks[0];
callBack.setName("EXAMPLE.COM");
}
});
login.config
SampleClient {
com.sun.security.auth.module.Krb5LoginModule required
default_realm=EXAMPLE.COM;
};
Adding following code before login worked.
try {
Configuration config = Configuration.getConfiguration();
config.getAppConfigurationEntry("SampleClient");
URIParameter uriParameter = new URIParameter(new java.net.URI("file:///path_to_your_file/login.conf"));
Configuration instance = Configuration.getInstance("JavaLoginConfig", uriParameter);
Configuration.setConfiguration(instance);
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}

Selenium - Exception while executing selenium.stop()

I am using Selenium for web application testing. After having finished all the testing, when the command "selenium.stop()" is executed I get the following exception:
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:760)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1049)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:373)
at com.thoughtworks.selenium.HttpCommandProcessor.getResponseCode(HttpCommandProcessor.java:134)
at com.thoughtworks.selenium.HttpCommandProcessor.getCommandResponseAsString(HttpCommandProcessor.java:154)
at com.thoughtworks.selenium.HttpCommandProcessor.executeCommandOnServlet(HttpCommandProcessor.java:97)
at com.thoughtworks.selenium.HttpCommandProcessor.doCommand(HttpCommandProcessor.java:79)
at com.thoughtworks.selenium.HttpCommandProcessor.stop(HttpCommandProcessor.java:242)
at com.thoughtworks.selenium.DefaultSelenium.stop(DefaultSelenium.java:111)
at seleniumproject.orkut.tearDown(orkut.java:54)
at junit.framework.TestCase.runBare(TestCase.java:140)
at com.thoughtworks.selenium.SeleneseTestCase.runBare(SeleneseTestCase.java:212)
at junit.framework.TestResult$1.protect(TestResult.java:110)
E
at junit.framework.TestResult.runProtected(TestResult.java:128)
Time: 63.453
There was 1 error:
1) testOrkut(seleniumproject.orkut)java.lang.UnsupportedOperationException: Catch body broken: IOException from cmd=testComplete -> java.net.ConnectException: Connection refused: connect
at com.thoughtworks.selenium.HttpCommandProcessor.executeCommandOnServlet(HttpCommandProcessor.java:100)
at com.thoughtworks.selenium.HttpCommandProcessor.doCommand(HttpCommandProcessor.java:79)
at com.thoughtworks.selenium.HttpCommandProcessor.stop(HttpCommandProcessor.java:242)
at com.thoughtworks.selenium.DefaultSelenium.stop(DefaultSelenium.java:111)
at seleniumproject.orkut.tearDown(orkut.java:54)
at com.thoughtworks.selenium.SeleneseTestCase.runBare(SeleneseTestCase.java:212)
at seleniumproject.orkut.main(orkut.java:57)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at junit.framework.TestResult.run(TestResult.java:113)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at junit.framework.TestCase.run(TestCase.java:124)
at java.net.Socket.connect(Socket.java:519)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at java.net.Socket.connect(Socket.java:469)
at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
at junit.framework.TestSuite.run(TestSuite.java:227)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at junit.textui.TestRunner.doRun(TestRunner.java:116)
at junit.textui.TestRunner.doRun(TestRunner.java:109)
at junit.textui.TestRunner.run(TestRunner.java:77)
at seleniumproject.orkut.main(orkut.java:57)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:760)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1049)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:373)
at com.thoughtworks.selenium.HttpCommandProcessor.getResponseCode(HttpCommandProcessor.java:134)
at com.thoughtworks.selenium.HttpCommandProcessor.getCommandResponseAsString(HttpCommandProcessor.java:154)
at com.thoughtworks.selenium.HttpCommandProcessor.executeCommandOnServlet(HttpCommandProcessor.java:97)
... 16 more
Here's is code which is executing the stop() command
public class orkut extends SeleneseTestCase {
public void setUp() throws Exception {
setUp("https://www.google.com/");
}
public void testOrkut() throws Exception {
Parser parser = new Parser();
try {
.......
.......
.......
}catch(Exception ex) {
}finally {
parser.closeConnection();
selenium.shutDownSeleniumServer();
}
}
public static Test suite() {
return new TestSuite(orkut.class);
}
public void tearDown(){
selenium.stop();
}
public static void main(String args[]) {
junit.textui.TestRunner.run(suite());
}
}
It looks like your shutting down the Selenium server before the command to close the browser session. Try commenting out the line with selenium.shutDownSeleniumServer();
I was getting the same error. I cleared my jar files and re-added them and the exception was gone.