I am trying to upload file to Dropbox using dropbox java sdk
I created a app in dropbox, gave permission to write, app status is Development.
The code works till "write mode done" logger, it fails after that. fileStream is of type InputStream
DbxClientV2 client = createClient();
String path = "/" + fileName.toString();
try {
System.out.println("STARTING...............");
UploadBuilder ub = client.files().uploadBuilder(path);
System.out.println("ub done");
ub.withMode(WriteMode.ADD);
System.out.println("write mode done");
FileMetadata metadata = ub.uploadAndFinish(fileStream);
System.out.println("File uploaded");
System.out.println(metadata.toStringMultiline());
;
System.out.println("FINISHED...............");
} catch (IOException e) {
e.printStackTrace();
}
Exception I get is
com.dropbox.core.BadResponseException: Bad JSON: expected object value.
at [Source: ; line: 1, column: 1]
at com.dropbox.core.DbxRequestUtil.unexpectedStatus(DbxRequestUtil.java:347)
at com.dropbox.core.DbxRequestUtil.unexpectedStatus(DbxRequestUtil.java:324)
at com.dropbox.core.DbxUploader.finish(DbxUploader.java:268)
at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:126)
.
.
.
Caused by: com.fasterxml.jackson.core.JsonParseException: expected object value.
at [Source: ; line: 1, column: 1]
at com.dropbox.core.stone.StoneSerializer.expectStartObject(StoneSerializer.java:91)
at com.dropbox.core.ApiErrorResponse$Serializer.deserialize(ApiErrorResponse.java:53)
Related
Application application_1552978163044_0016 failed 5 times due to AM Container for appattempt_1552978163044_0016_000005 exited with exitCode: -1000
Diagnostics:
java.io.IOException: Resource
abfs://xxx#xxx.dfs.core.windows.net/hdp/apps/2.6.5.3006-29/mapreduce/mapreduce.tar.gz
changed on src filesystem (expected 1552949440000, was 1552978240000
Failing this attempt. Failing the application.
Just based on the exception information, it seems to be caused by Azure Storage could not keep the original timestamp of the copied file. I searched a workaround that recommended to change the source code of yarn-common to disable the code block of timestamp check when copy file to avoid the exception throws to make the MR job continous to work.
Here is the source code in the latest version of yarn-common which check the timestamp for copied file and throws the exception.
/** #L255
* Localize files.
* #param destination destination directory
* #throws IOException cannot read or write file
* #throws YarnException subcommand returned an error
*/
private void verifyAndCopy(Path destination)
throws IOException, YarnException {
final Path sCopy;
try {
sCopy = resource.getResource().toPath();
} catch (URISyntaxException e) {
throw new IOException("Invalid resource", e);
}
FileSystem sourceFs = sCopy.getFileSystem(conf);
FileStatus sStat = sourceFs.getFileStatus(sCopy);
if (sStat.getModificationTime() != resource.getTimestamp()) {
throw new IOException("Resource " + sCopy +
" changed on src filesystem (expected " + resource.getTimestamp() +
", was " + sStat.getModificationTime());
}
if (resource.getVisibility() == LocalResourceVisibility.PUBLIC) {
if (!isPublic(sourceFs, sCopy, sStat, statCache)) {
throw new IOException("Resource " + sCopy +
" is not publicly accessible and as such cannot be part of the" +
" public cache.");
}
}
downloadAndUnpack(sCopy, destination);
}
I am use aws Lamda to decompress and traverse tar.gz files then uploading them back to s3 deflated retaining the original directory structure.
I am running into an issue streaming a TarArchiveEntry to a S3 bucket via a PutObjectRequest. While first entry is successfully streamed, upon trying to getNextTarEntry() on the TarArchiveInputStream a null pointer is thrown due to the underlying GunzipCompress inflator being null, which had an appropriate value prior to the s3.putObject(new PutObjectRequest(...)) call.
I have not been able to find documentation on how / why the gz input stream inflator attribute is being set to null after partially being sent to s3.
EDIT Further investigation has revealed that the AWS call appears to be closing the input stream after completing the upload of specified content length... haven't not been able to find how to prevent this behavior.
Below is essentially what my code looks like. Thank in advance for your help, comments, and suggestions.
public String handleRequest(S3Event s3Event, Context context) {
try {
S3Event.S3EventNotificationRecord s3EventRecord = s3Event.getRecords().get(0);
String s3Bucket = s3EventRecord.getS3().getBucket().getName();
// Object key may have spaces or unicode non-ASCII characters.
String srcKey = s3EventRecord.getS3().getObject().getKey();
System.out.println("Received valid request from bucket: " + bucketName + " with srckey: " + srcKeyInput);
String bucketFolder = srcKeyInput.substring(0, srcKeyInput.lastIndexOf('/') + 1);
System.out.println("File parent directory: " + bucketFolder);
final AmazonS3 s3Client = AmazonS3ClientBuilder.defaultClient();
TarArchiveInputStream tarInput = new TarArchiveInputStream(new GzipCompressorInputStream(getObjectContent(s3Client, bucketName, srcKeyInput)));
TarArchiveEntry currentEntry = tarInput.getNextTarEntry();
while (currentEntry != null) {
String fileName = currentEntry.getName();
System.out.println("For path = " + fileName);
// checking if looking at a file (vs a directory)
if (currentEntry.isFile()) {
System.out.println("Copying " + fileName + " to " + bucketFolder + fileName + " in bucket " + bucketName);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(currentEntry.getSize());
s3Client.putObject(new PutObjectRequest(bucketName, bucketFolder + fileName, tarInput, metadata)); // contents are properly and successfully sent to s3
System.out.println("Done!");
}
currentEntry = tarInput.getNextTarEntry(); // NPE here due underlying gz inflator is null;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
IOUtils.closeQuietly(tarInput);
}
}
That's true, AWS closes an InputStream provided to PutObjectRequest, and I don't know of a way to instruct AWS not to do so.
However, you can wrap the TarArchiveInputStream with a CloseShieldInputStream from Commons IO, like that:
InputStream shieldedInput = new CloseShieldInputStream(tarInput);
s3Client.putObject(new PutObjectRequest(bucketName, bucketFolder + fileName, shieldedInput, metadata));
When AWS closes the provided CloseShieldInputStream, the underlying TarArchiveInputStream will remain open.
PS. I don't know what ByteArrayInputStream(tarInput.getCurrentEntry()) does but it looks very strange. I ignored it for the purpose of this answer.
I am trying to use the following code to unload data into S3 bucket. Which works but after unloading it throws some error.
Properties props = new Properties();
props.setProperty("user", MasterUsername);
props.setProperty("password", MasterUserPassword);
conn = DriverManager.getConnection(dbURL, props);
stmt = conn.createStatement();
String sql;
sql = "unload('select * from part where p_partkey in (select p_partkey from
part limit 10)') to"
+ " 's3://redshiftdump.****' "
+ " DELIMITER AS ','"
+ "ADDQUOTES "
+ "NULL AS ''"
+ "credentials 'aws_access_key_id=****;aws_secret_access_key=***' "
+ "parallel off" +
";";
boolean i = stmt.execute(sql);
stmt.close();
conn.close();
The unloading works. It is creating a file in the bucket. But it is giving me some error
java.sql.SQLException:
dataengine.impl.DSISimpleRowCountResult cannot be cast to
com.amazon.dsi.dataengine.interfaces.IResultSet
at
com.amazon.redshift.core.jdbc42.PGJDBC42Statement.createResultSet(Unknown
Source)
at com.amazon.jdbc.common.SStatement.executeQuery(Unknown Source)
what is this error and how to avoid it? Is there any way to dump the table in CSV format. Right now it is dumping the file in FILE format.
You say the UNLOAD works but you receive this error, that suggests to me that you are connecting successfully but there is an problem in the way your code interacts with the JDBC driver when the query completes.
We provide an example that may be helpful in our documentation on the page "Connect to Your Cluster Programmatically"
Regarding the output file format, you will get whatever is specified in your UNLOAD SQL but the filename will have a suffix (for example "000" or "6411_part_00") to indicate which part of the UNLOAD it is.
use executeUpdate .
def runQuery(sql: String) = {
Class.forName("com.amazon.redshift.jdbc.Driver")
val connection = DriverManager.getConnection(url, username, password)
var statement: Statement = null
try {
statement = connection.createStatement()
statement.setQueryTimeout(redshiftTimeoutInSeconds)
val result = statement.executeUpdate(sql)
logger.info(s"statement response code : ${result}")
} catch {
case e: Exception => {
logger.error(s"statement.isCloseOnCompletion :${e.getMessage} ::: ${e.printStackTrace()}")
throw new IngestionException(e.getMessage)
}
}
finally {
if(statement != null ) statement.close()
connection.close()
}
}
I'm new to grails so hoping someone will be patient and give me a hand. I have a controller that creates a PDF. If the user clicks more then one time before the PDF is created I get the following error. Below is the code for the creation of the PDF.
2016-03-09 09:32:11,549 ERROR errors.GrailsExceptionResolver - SocketException occurred when processing request: [GET] /wetlands-form/assessment/f3458c91-3435-4714-a0e0-3b24de238671/assessment/pdf
Connection reset by peer: socket write error. Stacktrace follows:
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at mdt.wetlands.AssessmentController$_closure11$$EPeyAg3t.doCall(AssessmentController.groovy:300)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2016-03-09 09:32:11,549 ERROR errors.GrailsExceptionResolver - IllegalStateException occurred when processing request: [GET] /wetlands-form/assessment/f3458c91-3435-4714-a0e0-3b24de238671/assessment/pdf
getOutputStream() has already been called for this response. Stacktrace follows:
org.codehaus.groovy.grails.web.pages.exceptions.GroovyPagesException: Error processing GroovyPageView: getOutputStream() has already been called for this response
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException: getOutputStream() has already been called for this response
at C__MDTDATA_gg_workspace_new_wetlands_grails_app_views_error_gsp.run(error.gsp:1)
... 5 more
2016-03-09 09:32:11,549 ERROR [/wetlands-form].[grails] - Servlet.service() for servlet grails threw exception
java.lang.IllegalStateException: getOutputStream() has already been called for this response
PDF CODE VIA rendering plugin
def pdf = {
def assessment = lookupAssessment()
if (!assessment){
return
}
// Trac 219 Jasper report for PDF output
Map reportParams = [:]
def report = params.report
def printType = params.printType
def mitigationType = params.mitigationType
def fileName
def fileType
fileType = 'PDF'
def reportDir =
grailsApplication.mainContext.servletContext.getRealPath(""+File.separatorChar+"reports"+File.separatorChar)
def resolver = new SimpleFileResolver(new File(reportDir))
reportParams.put("ASSESS_ID", assessment.id)
reportParams.put("RUN_DIR", reportDir+File.separatorChar)
reportParams.put("JRParameter.REPORT_FILE_RESOLVER", resolver)
reportParams.put("_format", fileType)
reportParams.put("_file", "assessment")
println params
def reportDef = jasperService.buildReportDefinition(reportParams, request.getLocale(), [])
def file = jasperService.generateReport(reportDef).toByteArray()
// Non-inline reports (e.g. PDF)
if (!reportDef.fileFormat.inline && !reportDef.parameters._inline)
{
response.setContentType("APPLICATION/OCTET-STREAM")
response.setHeader("Content-disposition", "attachment; filename=" + assessment.name + "." + reportDef.fileFormat.extension);
response.contentType = reportDef.fileFormat.mimeTyp
response.characterEncoding = "UTF-8"
response.outputStream << reportDef.contentStream.toByteArray()
}
else
{
// Inline report (e.g. HTML)
render(text: reportDef.contentStream, contentType: reportDef.fileFormat.mimeTyp, encoding: reportDef.parameters.encoding ? reportDef.parameters.encoding : 'UTF-8');
}
}
This is the WORD code.
def word = {
def assessment = lookupAssessment()
if (!assessment){
return
}
// get the assessment's data as xml
def assessmentXml = g.render(template: 'word', model: [assessment:assessment]).toString()
// open the Word template
def loader = new LoadFromZipNG()
def template = servletContext.getResourceAsStream('/word/template.docx')
WordprocessingMLPackage wordMLPackage = (WordprocessingMLPackage)loader.get(template)
// get custom xml piece from Word template
String itemId = '{44f68b34-ffd4-4d43-b59d-c40f7b0a2880}' // have to pull up part by ID. Watch out - this may change if you muck with the template!
CustomXmlDataStoragePart customXmlDataStoragePart = wordMLPackage.getCustomXmlDataStorageParts().get(itemId)
CustomXmlDataStorage data = customXmlDataStoragePart.getData()
// and replace it with our assessment's xml
ByteArrayInputStream bs = new ByteArrayInputStream(assessmentXml.getBytes())
data.setDocument(bs) // needs java.io.InputStream
// that's it! the data is in the Word file
// but in order to do the highlighting, we have to manipulate the Word doc directly
// gather the list of cells to highlight
def highlights = assessment.highlights()
// get the main document from the Word file as xml
MainDocumentPart mainDocPart = wordMLPackage.getMainDocumentPart()
def xml = XmlUtils.marshaltoString(mainDocPart.getJaxbElement(), true)
// use the standard Groovy tools to handle the xml
def document = new XmlSlurper(keepWhitespace:true).parseText(xml)
// for each value in highlight list - find node, shade cell and add bold element
highlights.findAll{it != null}.each{highlight ->
def tableCell = document.body.tbl.tr.tc.find{it.sdt.sdtPr.alias.'#w:val' == highlight}
tableCell.tcPr.shd[0].replaceNode{
'w:shd'('w:fill': 'D9D9D9') // shade the cell
}
def textNodes = tableCell.sdt.sdtContent.p.r.rPr
textNodes.each{
it.appendNode{
'w:b'() // bold element
}
}
}
// here's a good way to print out xml for debugging
// System.out.println(new StreamingMarkupBuilder().bindNode(document.body.tbl.tr.tc.find{it.sdt.sdtPr.alias.#'w:val' == '12.1.1'}).toString())
// or save xml to file for study
// File testOut = new File("C:/MDTDATA/wetlands-trunk/xmlout.xml")
// testOut.setText(new StreamingMarkupBuilder().bindNode(document).toString())
// get the updated xml back in the Word doc
Object obj = XmlUtils.unmarshallFromTemplate(new StreamingMarkupBuilder().bindNode(document).toString(), null);
mainDocPart.setJaxbElement((Object)obj)
File file = File.createTempFile('wordexport-', '.docx')
wordMLPackage.save(file)
response.setHeader('Content-Type', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document;')
response.setHeader('Content-Disposition', "attachment; filename=${assessment.name.encodeAsURL()}.docx")
response.setHeader('Content-Length', "${file.size()}")
response.outputStream << file.readBytes()
response.outputStream.flush()
file.delete()
}
// for checking XML during development
def word2 = {
def assessment = lookupAssessment()
if (!assessment){
return
}
render template: 'word', model: [assessment:assessment]
}
You need to catch the exception, if you wish to not do anything with it then as below in the catch nothing going on.. after it has gone through try and catch if still no file we know something has gone wrong so we render another or same view with error this time. After this it returns so it won't continue to your other bit which checks report type i.e. pdf or html
..
//declare file (def means it could be any type of object)
def file
//Now when you expect unexpected behaviour capture it with a try/catch
try {
file = jasperService.generateReport(reportDef).toByteArray()
}catch (Exception e) {
//log.warn (e)
//println "${e} ${e.errors}"
}
//in your scenario or 2nd click the user will hit the catch segment
//and have no file produced that would be in the above try block
//this now says if file == null or if file == ''
// in groovy !file means capture if there nothing defined for file
if (!file) {
//render something else
render 'a message or return to page with error that its in use or something gone wrong'
//return tells your controller to stop what ever else from this point
return
}
//so what ever else would occur will not occur since no file was produced
...
Now a final note try/catches are expensive and should not be used everywhere. If you are expecting something then deal with the data. In scenarios typically like this third party api where you have no control i.e. to make the unexpected expected then you fall back to these methods
1- Client Side : Better is to disable button on first click and wait for response from Server.
2- Catch Exception and do nothing or just print error log.
// get/set parameters
def file
def reportDef
try{
reportDef = jasperService.buildReportDefinition(reportParams, request.getLocale(), [])
file = jasperService.generateReport(reportDef).toByteArray()
}catch(Exception e){
// print log or do nothing
}
if (file){
// render file according to your conditions
}
else {
// render , return appropriate message.
}
Instead of catching Exception, Its better to catch IOException. Otherwise you will be eating all other exceptions as well. Here is how i handled it.
private def streamFile(File file) {
def outputStream
try {
response.contentType = "application/pdf"
response.setHeader "Content-disposition", "inline; filename=${file.name}"
outputStream = response.outputStream
file.withInputStream {
response.contentLength = it.available()
outputStream << it
}
outputStream.flush()
}
catch (IOException e){
log.info 'Probably User Cancelled the download!'
}
finally {
if (outputStream != null){
try {
outputStream.close()
} catch (IOException e) {
log.info 'Exception on close'
}
}
}
}
The api is streaming large volume of data. after performing validation and opening up the connection to backend then creating the jdbc statement, we return httpresponse ok status with the header. The problem we see is that when the streaming breaks, the client does not get error code and only thing we can do is just close the channel.
Here is how we send the status back at the begining;
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
response.headers().set(CONTENT_TYPE, MimeTypes.TEXT_JSON_UTF_8);
response.setChunked(true);
response.headers().set(Names.TRANSFER_ENCODING, Values.CHUNKED);
Channel ch = ctx.getChannel();
// Write the initial line and the header.
ch.write(response);
When anything fails during the streaming, the error is captured by the catch block;
} catch (Exception e) {
ctx.getChannel().close();
String msg = "Error while streaming dynamic content from backend datasource " + ((DatasetDynamic) datasets[0]).getDbDataSourceName();
error(e, msg);
debug("uriNodes=" + this.uriNodes + "; params=" + this.params);
throw new Exception(msg, e);
} finally {
As you see in the catch block, to notify the client, something went wrong, all it is doing is;ctx.getChannel().close();
Is there anyway we can send proper httpresponse with error back to the client?
Looks like you can send httpresponse anytime thru the channel, it does not have to be as header;
} catch (Exception e) {
// Any exception here means an error in the middle of chunk streaming, we
// send back http 500 to the client to inform the failure
ResponseErrorStatus status = new ResponseErrorStatus(ServiceErrorStatus.INTERNAL_SERVER_ERROR,
" error: " + e.getMessage() + (e.getCause() != null ? (" - caused by: " + e.getCause().getMessage()) : ""));
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, status.serviceErrorStatus.httpStatus);
Channel ch = ctx.getChannel();
ch.write(response);