I have requirement to upload streaming data from java program to AWS S3. Below is my program. I don't know part size while uploading the stream.
if(!islastChunk)
uploadRequest.withPartSize(5242880);
else
uploadRequest.setLastPart(islastChunk);
So i have above logic to setPartSize as 5 MB when the lastPart is false and I set the setLastPart as true in UploadPartResult for last last part.
package com.aws.s3;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
import com.amazonaws.services.s3.model.PartETag;
import com.amazonaws.services.s3.model.UploadPartRequest;
import com.amazonaws.services.s3.model.UploadPartResult;
public class MultipartFileUpload {
public static void main(String[] args) {
AWSCredentials credentials = new BasicAWSCredentials("ABC", "DEF");
AmazonS3 s3 = new AmazonS3Client(credentials);
String existingBucketName = "streamdatastore";
String keyName = "PeddlerGuideData.txt";
List<PartETag> partETags = new ArrayList<PartETag>();
//String filePath = "";
int[] idx = { 1 };
InitiateMultipartUploadResult initResponse= initChunkDataUpload(s3,existingBucketName,keyName);
try {
String content = new String(Files.readAllBytes(Paths.get("MultiPartTestData.txt")));
InputStream is = new ByteArrayInputStream(content.getBytes());
uploadChunkData(s3,initResponse,initResponse.getBucketName(),initResponse.getKey(),1,is,partETags,false);
uploadChunkData(s3,initResponse,initResponse.getBucketName(),initResponse.getKey(),2,is,partETags,true);
} catch (IOException e) {
e.printStackTrace();
}
completeUploadData(s3,initResponse,initResponse.getBucketName(),initResponse.getKey(),partETags);
}
private static void processLine(String line,AmazonS3 s3,InitiateMultipartUploadResult initResponse,int partNumber,List<PartETag> partETags){
System.out.println("Line number "+ partNumber+"\n line = \n"+line);
InputStream is = new ByteArrayInputStream(line.getBytes());
uploadChunkData(s3,initResponse,initResponse.getBucketName(),initResponse.getKey(),partNumber,is,partETags,false);
}
private static void putMultiPartRequest(){
}
private static InitiateMultipartUploadResult initChunkDataUpload(AmazonS3 s3Client,String existingBucketName,String keyName){
InitiateMultipartUploadRequest initRequest = new
InitiateMultipartUploadRequest(existingBucketName, keyName);
InitiateMultipartUploadResult initResponse =
s3Client.initiateMultipartUpload(initRequest);
return initResponse;
}
private static List<PartETag> uploadChunkData(AmazonS3 s3Client,InitiateMultipartUploadResult initResponse ,String existingBucketName,String keyName,int partNumber,InputStream inputStream,List<PartETag> partETags,boolean islastChunk){
UploadPartRequest uploadRequest = null;
try{
uploadRequest = new UploadPartRequest()
.withBucketName(existingBucketName).withKey(keyName)
.withUploadId(initResponse.getUploadId()).withPartNumber(partNumber).withInputStream(inputStream);
if(!islastChunk)
uploadRequest.withPartSize(5242880);
else
uploadRequest.setLastPart(islastChunk);
UploadPartResult uploadPartResult =s3Client.uploadPart(uploadRequest);
partETags.add(uploadPartResult.getPartETag());
System.out.println("chunk data upload is completed");
}catch(Exception e){
e.printStackTrace();
}finally{
}
return partETags;
}
private static void completeUploadData (AmazonS3 s3Client,InitiateMultipartUploadResult initResponse ,String existingBucketName,String keyName,List<PartETag> partETags){
try{
CompleteMultipartUploadRequest compRequest = new
CompleteMultipartUploadRequest(
existingBucketName,
keyName,
initResponse.getUploadId(),
partETags);
s3Client.completeMultipartUpload(compRequest);
}catch(Exception e){
e.printStackTrace();
}finally{
}
}
}
When I set the part size as 5 MB, I always get the below exception.
com.amazonaws.services.s3.model.AmazonS3Exception: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. (Service: Amazon S3; Status Code: 400; Error Code: RequestTimeout; Request ID: 4B6405551FAABAFB), S3 Extended Request ID: v5RWy2tut1mEMzDZ1mwe0bFj6m9ho6eUihuO0p0EsccQtWv2814Ak18jtBNvxqDdt3AHy1otvTI=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1530)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1168)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:949)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:662)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:636)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:619)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:587)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:574)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:446)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4035)
at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3045)
at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3030)
at com.aws.s3.MultipartFileUpload.uploadChunkData(MultipartFileUpload.java:81)
at com.aws.s3.MultipartFileUpload.main(MultipartFileUpload.java:44)
When I remove then logic that I have mentioned above, I am getting below error.
com.amazonaws.services.s3.model.AmazonS3Exception: Your proposed upload is smaller than the minimum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooSmall; Request ID: 11A9870A168EDBE0), S3 Extended Request ID: 7OnccoS26Ijd4SKvBj9mVvlH4SZk4oloNouNS5gc/QyS4DQXm5A24/Ds96WRHdlkr76nAaPSxR4=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1530)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1168)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:949)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:662)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:636)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:619)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:587)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:574)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:446)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4035)
at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2806)
at com.aws.s3.MultipartFileUpload.completeUploadData(MultipartFileUpload.java:103)
at com.aws.s3.MultipartFileUpload.main(MultipartFileUpload.java:49)
Any suggestion on how to proceed with the same.
Look at or consider using of MultipartUploadOutputStream implemented in s3distcp library.
Related
I have written the following code to upload collection strings to S3 using multipart upload (each part is greater than 5 MB), but when I download it back from S3 it contains only the strings from one part.
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.*;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
public class LowLevelMultipartUpload {
private List<PartETag> partETags = new ArrayList<>();
private InitiateMultipartUploadResult initResponse;
AmazonS3 s3Client; String bucketName; String keyName;
public LowLevelMultipartUpload(AmazonS3 s3Client, String bucketName, String keyName) {
this.s3Client = s3Client;
this.bucketName = bucketName;
this.keyName = keyName;
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, keyName);
initResponse = s3Client.initiateMultipartUpload(initRequest);
}
public void uploadPart(int partNumber, InputStream is, ObjectMetadata objectMetadata, boolean isFinalPart) throws IOException {
// Create the request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(bucketName)
.withKey(keyName)
.withUploadId(initResponse.getUploadId())
.withPartNumber(partNumber)
.withInputStream(is)
.withPartSize(is.available())
.withObjectMetadata(objectMetadata);
if(isFinalPart) uploadRequest.withLastPart(true);
// Upload the part and add the response's ETag to our list.
UploadPartResult uploadResult = s3Client.uploadPart(uploadRequest);
partETags.add(uploadResult.getPartETag());
}
public void completeUpload() {
// Complete the multipart upload.
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, keyName,
initResponse.getUploadId(), partETags);
s3Client.completeMultipartUpload(compRequest);
}
}
I call the above code in the following order.
Create LowLevelMultipartUpload object
Upload parts in loop
Call complete upload
No Issues in the above code, It is working fine.
I have been using the rally javatoolkit for a while to add testcases, test results etc without any error. But all of a sudden it started throwing error as
" javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated" . I have referred the pages "rally rest api java toolkit sslpeerunverifiedexception : peer not authenticated" , rally rest api java toolkit sslpeerunverifiedexception : peer not authenticated but they didn't help me. Can someone help me with what I am doing wrong. Also If i need to download a certificate please help me for windows system. Thanks in advance. my code is as below:
import com.rallydev.rest.RallyRestApi;
import com.rallydev.rest.client.HttpClient;
import com.rallydev.rest.request.GetRequest;
import com.rallydev.rest.response.GetResponse;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.conn.scheme.Scheme;
public class ConnnectionTestWithHTTPClient {
public static void main(String[] args) throws URISyntaxException, IOException {
String host = "https://rally1.rallydev.com";
String apiKey = "_abc123";
String applicationName = "Connnection Test With HTTPClient";
RallyRestApi restApi = new RallyRestApi(new URI(host),apiKey);
restApi.setApplicationName(applicationName);
//restApi.setProxy(new URI("http://myproxy.mycompany.com"), "MyProxyUsername", "MyProxyPassword"); //SET PROXY SETTINS HERE
HttpClient client = restApi.getClient();
try {
SSLSocketFactory sf = new SSLSocketFactory(new TrustStrategy() {
public boolean isTrusted(X509Certificate[] certificate, String authType)
throws CertificateException {
//trust all certs
return true;
}
}, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
client.getConnectionManager().getSchemeRegistry().register(new Scheme("https", 443, sf));
String workspaceRef = "/workspace/12345"; //USE VALID WORKSPACE OID
GetRequest getRequest = new GetRequest(workspaceRef);
GetResponse getResponse = restApi.get(getRequest);
System.out.println(getResponse.getObject());
} catch (Exception e) {
System.out.println(e);
} finally {
restApi.close();
}
}
}
Also adding to the issue, i found a different error when I changed the port from 443 to 8443. i get "java.io.IOException: HTTP/1.1 522 Origin Connection Time-out" when i use 8443.
For some reason when I uncomment the line //restApi.setProxy(new URI("http://myproxy.mycompany.com"), "MyProxyUsername", "MyProxyPassword"); with correct inputs, the error goes off.
For all those who need the inputs, please put in the following:
restApi.setProxy(new URI("http://rally1.rallydev.com"), "xyz#abc.com", "rallypassword");
so the working code is as below:
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.conn.ssl.TrustStrategy;
import com.rallydev.rest.RallyRestApi;
import com.rallydev.rest.client.HttpClient;
import com.rallydev.rest.request.GetRequest;
import com.rallydev.rest.response.GetResponse;
public class ConnnectionTestWithHTTPClient {
public static void main(String[] args) throws URISyntaxException, IOException {
String host = "https://rally1.rallydev.com";
String apiKey = "_apikey";
String applicationName = "Connnection Test With HTTPClient";
RallyRestApi restApi = new RallyRestApi(new URI(host),apiKey);
restApi.setApplicationName(applicationName);
restApi.setProxy(new URI("http://rally1.rallydev.com"), "abc#abc.com", "rallypassword"); //YOUR PROXY SETTINGS HERE
HttpClient client = restApi.getClient();
try {
SSLSocketFactory sf = new SSLSocketFactory(new TrustStrategy() {
public boolean isTrusted(X509Certificate[] certificate, String authType)
throws CertificateException {
//trust all certs
return true;
}
}, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
client.getConnectionManager().getSchemeRegistry().register(new Scheme("https", 443, sf));
String workspaceRef = "/workspace/1234";
GetRequest getRequest = new GetRequest(workspaceRef);
GetResponse getResponse = restApi.get(getRequest);
System.out.println(getResponse.getObject());
} catch (Exception e) {
System.out.println(e);
} finally {
restApi.close();
}
}
}
We are experiencing problems while working with Swisscom S3 Dynamic Storage.
When making concurrent test CRUD requests in 5 and more parrallel threads, Storage service sends us randomly 403 Forbidden instead of corrent answer. While executing same requests sequentially, one-by-one, everything works ok.
Code that I am using is here below
import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import com.amazonaws.util.StringInputStream;
import org.apache.commons.io.IOUtils;
import org.junit.Test;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
/**
* Tutorial https://javatutorial.net/java-s3-example
*/
public class AmazonS3ManualTest {
public static final String BUCKET_NAME = "??";
private static String accessKey = "??";
private static String secretKey = "??";
#Test
public void testOperations() throws IOException, InterruptedException {
final int maxCount = 5;
final AmazonS3Client amazonS3Client = getS3Client();
final CountDownLatch latch = new CountDownLatch(maxCount);
final ExecutorService executor = Executors.newFixedThreadPool(maxCount);
for (int i = 0; i < maxCount; i++) {
final int index = i;
executor.submit(() -> {
try {
final String FolderOne = "testFolderOne" + index;
final String FolderTwo = "testFolderTwo" + index;
final String FolderCopy = "copyFolder" + index;
try {
createFile(amazonS3Client, "/" + FolderOne + "/file.txt");
createFolder(amazonS3Client, FolderTwo + "/");
exists(amazonS3Client, FolderOne + "/file.txt");
exists(amazonS3Client, FolderTwo + "/");
copy(amazonS3Client, FolderOne + "/file.txt", FolderCopy + "/filecopy.txt");
delete(amazonS3Client, "/" + FolderOne);
delete(amazonS3Client, "/" + FolderTwo);
get(amazonS3Client, FolderCopy + "/filecopy.txt");
delete(amazonS3Client, "/" + FolderCopy + "/filecopy.txt");
isEmptyFolder(amazonS3Client, "/" + FolderCopy);
delete(amazonS3Client, "/ + FolderCopy");
} catch (Exception e) {
e.printStackTrace();
}
latch.countDown();
} catch (final Exception ignored) {
}
});
}
if (!latch.await(300, TimeUnit.SECONDS)) {
throw new RuntimeException("Waiting too long for the result");
}
}
private void isEmptyFolder(AmazonS3Client amazonS3Client, String folder) {
final ObjectListing objectListing = amazonS3Client.listObjects(BUCKET_NAME, folder);
assert(objectListing.getObjectSummaries().isEmpty());
}
private void get(AmazonS3Client amazonS3Client, String file) throws IOException {
GetObjectRequest request = new GetObjectRequest(BUCKET_NAME, file);
final S3Object object = amazonS3Client.getObject(request);
final S3ObjectInputStream objectContent = object.getObjectContent();
final String s = IOUtils.toString(objectContent);
assert(s.length() > 0);
}
private void copy(AmazonS3Client amazonS3Client, String source, String target) {
CopyObjectRequest request = new CopyObjectRequest(BUCKET_NAME, source, BUCKET_NAME, target);
amazonS3Client.copyObject(request);
}
private void delete(AmazonS3Client amazonS3Client, String path) {
deleteRecursive(amazonS3Client, path);
}
private void deleteRecursive(AmazonS3Client amazonS3Client, String path) {
ObjectListing objects = amazonS3Client.listObjects(BUCKET_NAME, path);
for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) {
if (objectSummary.getKey().equals(path)) {
continue;
}
if (objectSummary.getKey().endsWith("/")) {
deleteRecursive(amazonS3Client, objectSummary.getKey());
} else {
amazonS3Client.deleteObject(BUCKET_NAME, objectSummary.getKey());
}
}
amazonS3Client.deleteObject(BUCKET_NAME, path);
}
private void exists(AmazonS3Client amazonS3Client, String folder) {
GetObjectMetadataRequest request = new GetObjectMetadataRequest(BUCKET_NAME, folder);
try {
final ObjectMetadata objectMetadata = amazonS3Client.getObjectMetadata(request);
assert(objectMetadata != null);
} catch (AmazonS3Exception e) {
if (e.getMessage().contains("404")) {
assert(false);
return;
}
}
assert(true);
}
private void createFolder(AmazonS3Client amazonS3Client, String folder) {
final InputStream input = new ByteArrayInputStream(new byte[0]);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(0);
amazonS3Client.putObject(new PutObjectRequest(BUCKET_NAME, folder, input, metadata));
}
private void createFile(AmazonS3Client amazonS3Client, String fileName) throws IOException {
ObjectMetadata omd = new ObjectMetadata();
//omd.setContentType("html/text");
omd.setHeader("filename", fileName);
omd.setHeader("x-amz-server-side-encryption", "AES256");
// upload file to folder and set it to public
final StringInputStream testFile = new StringInputStream("Test");
final PutObjectRequest putObjectRequest = new PutObjectRequest(BUCKET_NAME, fileName, testFile, omd);
amazonS3Client.putObject(putObjectRequest.withCannedAcl(CannedAccessControlList.Private));
testFile.close();
}
private AmazonS3Client getS3Client() {
ClientConfiguration opts = new ClientConfiguration();
opts.setSignerOverride("S3SignerType"); // NOT "AWS3SignerType"
opts.setMaxConnections(100);
final AmazonS3Client s3 = new AmazonS3Client(new BasicAWSCredentials(accessKey, secretKey), opts);
s3.setEndpoint("ds31s3.swisscom.com");
return s3;
}
}
Exception we are getting is here:
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: null), S3 Extended Request ID: null
Can you please recommend, what can we do with such a situation, because it's abnormal and not scalable.
I recreated new Dynamic Storage S3 and rerun test above. Now exceptions were not raising. Seems there was infra problem with previously created Storage.
We run your snippet 80 times in a row against the S3 Dynamic Storage of Swisscom and couldn't reproduce the issue.
However there might be a timing issue when directly accessing the object after uploading it. The PUT request may be balanced to another node than the GET request. So if you immediately want to download an object after uploading it, please implement a short sleep or a retry.
I am getting the following error:
Exception in thread "main" java.lang.Error: Unresolved compilation problem: The method newInstance(FopFactoryConfig) in the type FopFactory is not applicable for the arguments ()
at fopdemo.fopvass.PDFHandler.createPDFFile(PDFHandler.java:42)
at fopdemo.fopvass.TestPDF.main(TestPDF.java:40)
This is my code:
package fopdemo.fopvass;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.URL;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.transform.Result;
import javax.xml.transform.Source;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.TransformerFactoryConfigurationError;
import javax.xml.transform.dom.DOMResult;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.sax.SAXResult;
import javax.xml.transform.stream.StreamSource;
import org.apache.fop.apps.FOPException;
import org.apache.fop.apps.FOUserAgent;
import org.apache.fop.apps.Fop;
import org.apache.fop.apps.FopFactory;
import org.apache.fop.apps.MimeConstants;
public class PDFHandler {
public static final String EXTENSION = ".pdf";
public String PRESCRIPTION_URL = "template.xsl";
public String createPDFFile(ByteArrayOutputStream xmlSource, String templateFilePath) throws IOException {
File file = File.createTempFile("" + System.currentTimeMillis(), EXTENSION);
URL url = new File(templateFilePath + PRESCRIPTION_URL).toURI().toURL();
// creation of transform source
StreamSource transformSource = new StreamSource(url.openStream());
// create an instance of fop factory
FopFactory fopFactory = FopFactory.newInstance();
// a user agent is needed for transformation
FOUserAgent foUserAgent = fopFactory.newFOUserAgent();
// to store output
ByteArrayOutputStream pdfoutStream = new ByteArrayOutputStream();
StreamSource source = new StreamSource(new ByteArrayInputStream(xmlSource.toByteArray()));
Transformer xslfoTransformer;
try {
TransformerFactory transfact = TransformerFactory.newInstance();
xslfoTransformer = transfact.newTransformer(transformSource);
// Construct fop with desired output format
Fop fop;
try {
fop = fopFactory.newFop(MimeConstants.MIME_PDF, foUserAgent, pdfoutStream);
// Resulting SAX events (the generated FO)
// must be piped through to FOP
Result res = new SAXResult(fop.getDefaultHandler());
// Start XSLT transformation and FOP processing
try {
// everything will happen here..
xslfoTransformer.transform(source, res);
// if you want to save PDF file use the following code
OutputStream out = new java.io.FileOutputStream(file);
out = new java.io.BufferedOutputStream(out);
FileOutputStream str = new FileOutputStream(file);
str.write(pdfoutStream.toByteArray());
str.close();
out.close();
} catch (TransformerException e) {
e.printStackTrace();
}
} catch (FOPException e) {
e.printStackTrace();
}
} catch (TransformerConfigurationException e) {
e.printStackTrace();
} catch (TransformerFactoryConfigurationError e) {
e.printStackTrace();
}
return file.getPath();
}
public ByteArrayOutputStream getXMLSource(EmployeeData data) throws Exception {
JAXBContext context;
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
try {
context = JAXBContext.newInstance(EmployeeData.class);
Marshaller m = context.createMarshaller();
m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
m.marshal(data, System.out);
m.marshal(data, outStream);
} catch (JAXBException e) {
e.printStackTrace();
}
return outStream;
}
}
This is where it is called:
package fopdemo.fopvass;
import java.io.ByteArrayOutputStream;
import java.util.ArrayList;
/**
* #author Debasmita.Sahoo
*
*/
public class TestPDF {
public static void main(String args[]){
System.out.println("Hi Testing");
ArrayList employeeList = new ArrayList();
String templateFilePath ="C:/Paula/Proyectos/fop/fopvass/resources/";
Employee e1= new Employee();
e1.setName("Debasmita1 Sahoo");
e1.setEmployeeId("10001");
e1.setAddress("Pune");
employeeList.add(e1);
Employee e2= new Employee();
e2.setName("Debasmita2 Sahoo");
e2.setEmployeeId("10002");
e2.setAddress("Test");
employeeList.add(e2);
Employee e3= new Employee();
e3.setName("Debasmita3 Sahoo");
e3.setEmployeeId("10003");
e3.setAddress("Mumbai");
employeeList.add(e3);
EmployeeData data = new EmployeeData();
data.setEemployeeList(employeeList);
PDFHandler handler = new PDFHandler();
try {
ByteArrayOutputStream streamSource = handler.getXMLSource(data);
handler.createPDFFile(streamSource,templateFilePath);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
The following line won't compile:
FopFactory fopFactory = FopFactory.newInstance();
The method newInstance() requires a parameter. From the documentation (provided by Mathias Muller), under 'Basic Usage', that parameter refers to a configuration file:
FopFactory fopFactory = FopFactory.newInstance(
new File( "C:/Temp/fop.xconf" ) );
You need to provide it a File object. Alternatively, it is also possible to create a FopFactory instance by providing a URI to resolve relative URIs in the input file (because SVG files reference other SVG files). As the documentation suggests:
FopFactory fopFactory = FopFactory.newInstance(
new File(".").toURI() );
The user agent is the entity that allows you to interact with a single rendering run, i.e. the processing of a single document. If you wish to customize the user agent's behaviour, the first step is to create your own instance of FOUserAgent using the appropriate factory method on FopFactory and pass that to the factory method that will create a new Fop instance.
FopFactory.newInstance() support
FopFactory.newInstance(File)
FopFactory.newInstance(Uri)
FopFactory.newInstance(Uri, InputStream)
FopFactory.newInstance(FopFactoryConfig)
I successfully implemented on Windows and Mac OS (i.e. a Unix\Linux like NFS) the following code
URI uri = new File(config.getRoot()).toPath().toUri();
FopFactory fopFactory = FopFactory.newInstance(uri);
On the other hand, this does not work (and I don't understand why)
URI uri = new File(config.getRoot()).toUri();
I would like to implement push notifications for my website (obviously only in compatible browser as Safari 7).
I have read the Apple documentation and I have successfully created my package containing my icon.iconset, my certificate.p12, manifest.json and a website.json.
Now I would like to ask the permission to the user when I first visit the website. If he allows it, I should send the package.
Everything is pretty clear but I don't know how to go on.
How do I create my push package out of my files? How do I precisely sign it? The package should be always the same so I could sign it on my mac and upload to my server only one package.
If you have experience with this technology, please let me know :)
I have successfully create push package to ask permission on safari web push notifications using REST API in java. Also I have follow steps which provide by Apple officials on there sites.
Please follow below steps to create push package.
Create your web push notification P12 certificate from your apple account.
Create your REST API using https for pushPackage which contain icon.iconset, your certificate.p12, manifest.json and a website.json. p12 certificate must be web push notification certificate.
How to create push package:- Please refer below java code. I have create push package using java servlet. which provide 2 web service end points.
v1/pushpackage/webpushID
v1/log
Servlet which process your push package request which send by safari push notification method
SafariPushPakageAPI.java
/*
Push Package REST API handler
*/
import java.io.IOException;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Map;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang.StringUtils;
import com.safari.Packager;
public class SafariPushPakageAPI extends HttpServlet{
/**
*
*/
private static final long serialVersionUID = 1L;
public static String ServerPath = null;
private static final String REQUEST_PERMISSION = "/v1/pushPackages/YOUR_WEB_PUSH_ID";
private static final String REQUEST_ERRORLOG = "/v1/log";
#Override
public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
doRequest(request, response);
}
// /v1/pushPackages/webpushID
// /v1/log
#Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
doRequest(request, response);
}
private void doRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
System.out.println("===>> SAFARI PUSH NOTIFICATION REQUEST");
String path = request.getPathInfo();
System.out.println("PATH ===>> "+path);
if(path == null){
doRequestPermission(request, response);
}else if (path.equalsIgnoreCase(REQUEST_PERMISSION)){
doRequestPermission(request, response);
}else if (path.equalsIgnoreCase(REQUEST_ERRORLOG)){
doRequestShowErrorLog(request, response);
}else{
doRequestPermission(request, response);
}
}
private void doRequestPermission(HttpServletRequest request,HttpServletResponse response) {
try{
System.out.println("INSIDE REQUEST PERMISSION ==>>>");
System.out.println(IOUtils.toString(request.getReader()));
String authToken = StringUtils.isBlank(request.getParameter("token")) ? "UserTokenRT124DFGH" : StringUtils.trimToEmpty(request.getParameter("token"));
System.out.println("=>>>>>>>>>> USER TOKEN =>>>>>>>>>> "+authToken);
#SuppressWarnings("deprecation")
String packagePath =request.getRealPath("pushPackage.raw/icon.iconset/"); // LOCATION WHERE YOUR PUSH PACKAGE FOLDER CONTAIN LOGOS AND website.json file
response.setContentType("application/zip");
response.setHeader("Content-Disposition", "attachment;filename=\"pushpackage.zip\"");
OutputStream out = response.getOutputStream();
out.write(Packager.createPackageFile(authToken,packagePath));
response.flushBuffer();
}catch(IOException ioe){
ioe.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
private void doRequestShowErrorLog(HttpServletRequest request,HttpServletResponse response) {
try{
System.out.println("ERROR LOG STARTED");
System.out.println(IOUtils.toString(request.getReader()));
System.out.println("END");
}catch(Exception e){
e.printStackTrace();
}
}
}
Packager.java
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.io.IOUtils;
import org.json.JSONArray;
import org.json.JSONObject;
/**
*
* #author Ritesh
*/
public class Packager {
final static String CERTIFICATE_PATH="PATH TO YOUR 12 CERTIFICATE";
final static String CERTIFICATE_PASS="PASSWORD";
static String getJSON(String authenticationToken) throws Exception {
JSONObject obj = new JSONObject();
obj.put("websiteName", "WEB SITE NAME");
obj.put("websitePushID", "WEB PUSH ID");
obj.put("allowedDomains", new JSONArray());
obj.getJSONArray("allowedDomains").put("https://TEST.EXAMPLE.net");//LIST OF DOMAINS ALLOW
obj.put("urlFormatString", "https://TEST.EXAMPLE.net/%#");
obj.put("authenticationToken", authenticationToken);
obj.put("webServiceURL", "https://API.EXAMPLE.COM");//callback URL WITHOUT WEB SERVICE ENDPOINT NAME
return obj.toString();
}
public static byte[] createPackageFile(String authenticationToken, String path) throws Exception {
System.out.println("packaging safari file with token: " + authenticationToken);
ZipHandler zip = new ZipHandler();
File dir = new File(path);
for (File file : dir.listFiles()) {
InputStream is = new FileInputStream(file);
byte[] bytes = IOUtils.toByteArray(is);
zip.addFile("icon.iconset", file.getName(),bytes );
}
zip.addFile("", "website.json", getJSON(authenticationToken).getBytes());
byte[] manifest = zip.manifest();
zip.addFile("", "manifest.json", manifest);
zip.addFile("", "signature", sign(manifest));
return zip.getBytes();
}
static byte[] sign(byte bytesToSign[]) throws Exception {
return new PKCS7Signer().sign(CERTIFICATE_PATH,CERTIFICATE_PASS, bytesToSign);
}
/**
* Servlet handler , should listen on the callback URL (as in webServiceURL)
* #param requestPath
* #param req
* #param servletRequest
* #param servletResponse
* #throws Exception
*/
public static void main(String[] args) throws Exception {
Packager.createPackageFile("SafriNotifcation","");
}
}
PKCS7Signer.java which create your signature file.
import java.io.FileInputStream;
import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.Security;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.List;
import org.bouncycastle.cert.X509CertificateHolder;
import org.bouncycastle.cert.jcajce.JcaCertStore;
import org.bouncycastle.cms.CMSProcessableByteArray;
import org.bouncycastle.cms.CMSSignedData;
import org.bouncycastle.cms.CMSSignedDataGenerator;
import org.bouncycastle.cms.CMSTypedData;
import org.bouncycastle.cms.jcajce.JcaSignerInfoGeneratorBuilder;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.operator.ContentSigner;
import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
import org.bouncycastle.operator.jcajce.JcaDigestCalculatorProviderBuilder;
import org.bouncycastle.util.Store;
public final class PKCS7Signer {
static {
try{
Security.addProvider(new BouncyCastleProvider());
}catch(Exception e){
e.printStackTrace();
}
}
private KeyStore getKeystore(String storeLocation, String storePasswd) throws Exception {
if (storeLocation == null) {
System.out.println("Could not find store file (.p12)");
return null;
}
// First load the keystore object by providing the p12 file path
KeyStore clientStore = KeyStore.getInstance("PKCS12");
// replace testPass with the p12 password/pin
clientStore.load(new FileInputStream(storeLocation), storePasswd.toCharArray());
return clientStore;
}
private X509CertificateHolder getCert(KeyStore keystore, String alias) throws Exception {
java.security.cert.Certificate c = keystore.getCertificate(alias);
return new X509CertificateHolder(c.getEncoded());
}
private PrivateKey getPrivateKey(KeyStore keystore, String alias, String storePasswd) throws Exception {
return (PrivateKey) keystore.getKey(alias, storePasswd.toCharArray());
}
public byte[] sign(String storeLocation, String storePasswd, byte[] dataToSign) throws Exception {
KeyStore clientStore = getKeystore(storeLocation, storePasswd);
if (clientStore == null) {
return null;
}
Enumeration aliases = clientStore.aliases();
String alias = "";
while (aliases.hasMoreElements()) {
alias = (String) aliases.nextElement();
if (clientStore.isKeyEntry(alias)) {
break;
}
}
CMSTypedData msg = new CMSProcessableByteArray(dataToSign); // Data to sign
X509CertificateHolder x509Certificate = getCert(clientStore, alias);
List certList = new ArrayList();
certList.add(x509Certificate); // Adding the X509 Certificate
Store certs = new JcaCertStore(certList);
CMSSignedDataGenerator gen = new CMSSignedDataGenerator();
// Initializing the the BC's Signer
ContentSigner sha1Signer = new JcaContentSignerBuilder("SHA1withRSA").setProvider("BC").build(
getPrivateKey(clientStore, alias, storePasswd));
gen.addSignerInfoGenerator(new JcaSignerInfoGeneratorBuilder(new JcaDigestCalculatorProviderBuilder()
.setProvider("BC").build()).build(sha1Signer, x509Certificate));
// adding the certificate
gen.addCertificates(certs);
// Getting the signed data
CMSSignedData sigData = gen.generate(msg, false);
return sigData.getEncoded();
}
}
please note used latest bouncy castle jars for create signature file
bcprov-jdk15on-157.jar
bcpkix-jdk15on-157.jar
Write your javascript on your client side page.it contains simple java script to get permissions.
///
// For safari
var domain="YOUR WEB PUSH ID";
function safariIniti() {
var pResult = window.safari.pushNotification.permission(domain);
if(pResult.permission === 'default') {
//request permission
requestPermissions();
} else if (pResult.permission === 'granted') {
console.log("Permission for " + domain + " is " + pResult.permission);
var token = pResult.deviceToken;
// Show subscription for debug
console.log('Subscription details:'+token);
} else if(pResult.permission === 'denied') {
console.log("Permission for " + domain + " is " + pResult.permission);
}
}
function getToken(){
// always start with a letter (for DOM friendlyness)
var idstr=String.fromCharCode(Math.floor((Math.random()*25)+65));
do {
// between numbers and characters (48 is 0 and 90 is Z (42-48 = 90)
var ascicode=Math.floor((Math.random()*42)+48);
if (ascicode<58 || ascicode>64){
// exclude all chars between : (58) and # (64)
idstr+=String.fromCharCode(ascicode);
}
} while (idstr.length<32);
return (idstr);
}
function requestPermissions() {
var tokenVal = getToken();
window.safari.pushNotification.requestPermission('WEb service url without end points',domain,{token:tokenVal},
function(subscription) {
console.log(subscription.permission);
console.log("PERMISSION ====>> "+subscription.permission);
if(subscription.permission === 'granted') {
//TODO
}
else if(subscription.permission === 'denied') {
// TODO:
}
});
}
Apple provides a php file that you can use to create your push package including the signature. https://developer.apple.com/library/mac/documentation/NetworkingInternet/Conceptual/NotificationProgrammingGuideForWebsites/CompanionFile.zip
Alternatively, you can use the push_package gem, https://github.com/SymmetricInfinity/push_package, which we developed when implementing safari push notifications for zeropush.com. More details are available at https://zeropush.com/blog/implementing-safari-push-notifications-in-osx-mavericks.
Follow this apple documentation and github repo, they include sufficient information required to create a safari push notifications.