Dart file copy in Windows - file-io

The File class in the dart:io library doesn't yet include copy() and move() methods.
To tide me over until they arrive, I'm trying to roll my own copy function. I'm using the code below on Windows, but it just creates a 0kb file.
void copyFile(String input, String output) {
var inFile = new File(input), outFile = new File(output);
if (outFile.existsSync()) outFile.deleteSync(); // I realize this isn't required
var inStream = null, outStream = null;
try {
inStream = inFile.openInputStream();
outStream = outFile.openOutputStream(FileMode.WRITE);
inStream.pipe(outStream);
} finally {
if (outStream != null && !outStream.closed) outStream.close();
if (inStream != null && !inStream.closed) inStream.close();
}
}
I've also tried replacing the pipe line with print(inStream.read(100).toString()); and I get null. The input file does exist (otherwise I'd get a FileIOException). Am I doing something wrong, or are input streams broken under Windows?
I'm using:
Dart Editor version 0.3.1_r17463
Dart SDK version 0.3.1.2_r17463
Edit: The following works (although it doesn't "chunk"). Am I using the streams above incorrectly?
void copyFile(String input, String output) {
var inFile = new File(input), outFile = new File(output);
if (outFile.existsSync()) outFile.deleteSync(); // I realize this isn't required
outFile.writeAsBytesSync(inFile.readAsBytesSync(), FileMode.WRITE);
}

With your first code snippet, you get an empty file because pipe is not a synchronous method. Thus, the copy of inputStream to outputStream has not started when the finally block is execute. By closing the streams in this finally block, you stop the pipe before it even starts. Without that finally block the copy is done correctly.
void copyFile(String input, String output) {
final inStream = new File(input).openInputStream();
final outStream = new File(output).openOutputStream(FileMode.WRITE);
inStream.pipe(outStream);
}
Finally, you don't have to worry about closing streams because pipe close streams by default once achieved. See InputStream.pipe.

For synchronous copy, use:
File(sourceFile).copySync(destinationFile);
For asynchronous copy, use:
File(sourceFile).copy(destinationFile);

Related

Error using OpenXML to read a .docx file from a memorystream to a WordprocessingDocument to a string and back

I have an existing library that I can use to receive a docx file and return it. The software is .Net Core hosted in a Linux Docker container.
It's very limited in scope though and I need to perform some actions it can't do. As these are straightforward I thought I would use OpenXML, and for my proof of concept all I need to do is to read a docx as a memorystream, replace some text, turn it back into a memorystream and return it.
However the docx that gets returned is unreadable. I've commented out the text replacement below to eliminate that, and if I comment out the call to the method below then the docx can be read so I'm sure the issue is in this method.
Presumably I'm doing something fundamentally wrong here but after a few hours googling and playing around with the code I am not sure how to correct this; any ideas what I have wrong?
Thanks for the help
private MemoryStream SearchAndReplace(MemoryStream mem)
{
mem.Position = 0;
using (WordprocessingDocument wordDoc = WordprocessingDocument.Open(mem, true))
{
string docText = null;
StreamReader sr = new StreamReader(wordDoc.MainDocumentPart.GetStream());
docText = sr.ReadToEnd();
//Regex regexText = new Regex("Hello world!");
//docText = regexText.Replace(docText, "Hi Everyone!");
MemoryStream newMem = new MemoryStream();
newMem.Position = 0;
StreamWriter sw = new StreamWriter(newMem);
sw.Write(docText);
return newMem;
}
}
If your real requirement is to search and replace text in a WordprocessingDocument, you should have a look at this answer.
The following unit test shows how you can make your approach work if the use case really demands that you read a string from a part, "massage" the string, and write the changed string back to the part. It also shows one of the shortcomings of any other approach than the one described in the answer already mentioned above, e.g., by demonstrating that the string "Hello world!" will not be found in this way if it is split across w:r elements.
[Fact]
public void CanSearchAndReplaceStringInOpenXmlPartAlthoughThisIsNotTheWayToSearchAndReplaceText()
{
// Arrange.
using var docxStream = new MemoryStream();
using (var wordDocument = WordprocessingDocument.Create(docxStream, WordprocessingDocumentType.Document))
{
MainDocumentPart part = wordDocument.AddMainDocumentPart();
var p1 = new Paragraph(
new Run(
new Text("Hello world!")));
var p2 = new Paragraph(
new Run(
new Text("Hello ") { Space = SpaceProcessingModeValues.Preserve }),
new Run(
new Text("world!")));
part.Document = new Document(new Body(p1, p2));
Assert.Equal("Hello world!", p1.InnerText);
Assert.Equal("Hello world!", p2.InnerText);
}
// Act.
SearchAndReplace(docxStream);
// Assert.
using (WordprocessingDocument wordDocument = WordprocessingDocument.Open(docxStream, false))
{
MainDocumentPart part = wordDocument.MainDocumentPart;
Paragraph p1 = part.Document.Descendants<Paragraph>().First();
Paragraph p2 = part.Document.Descendants<Paragraph>().Last();
Assert.Equal("Hi Everyone!", p1.InnerText);
Assert.Equal("Hello world!", p2.InnerText);
}
}
private static void SearchAndReplace(MemoryStream docxStream)
{
using (WordprocessingDocument wordDocument = WordprocessingDocument.Open(docxStream, true))
{
// If you wanted to read the part's contents as text, this is how you
// would do it.
string partText = ReadPartText(wordDocument.MainDocumentPart);
// Note that this is not the way in which you should search and replace
// text in Open XML documents. The text might be split across multiple
// w:r elements, so you would not find the text in that case.
var regex = new Regex("Hello world!");
partText = regex.Replace(partText, "Hi Everyone!");
// If you wanted to write changed text back to the part, this is how
// you would do it.
WritePartText(wordDocument.MainDocumentPart, partText);
}
docxStream.Seek(0, SeekOrigin.Begin);
}
private static string ReadPartText(OpenXmlPart part)
{
using Stream partStream = part.GetStream(FileMode.OpenOrCreate, FileAccess.Read);
using var sr = new StreamReader(partStream);
return sr.ReadToEnd();
}
private static void WritePartText(OpenXmlPart part, string text)
{
using Stream partStream = part.GetStream(FileMode.Create, FileAccess.Write);
using var sw = new StreamWriter(partStream);
sw.Write(text);
}

missing blob with jgit

I am trying the following with jgit:
val git = Git.open(File("/path/toMyRepo"))
val diffFormatter = DiffFormatter(DisabledOutputStream.INSTANCE).apply {
setRepository(git.repository)
}
git.diff().call().forEach {
if (it.changeType == DiffEntry.ChangeType.MODIFY) {
diffFormatter.toFileHeader(it).toEditList().forEach {
println(it)
}
}
}
but I am getting a the following exception:
"org.eclipse.jgit.errors.MissingObjectException: Missing blob 9645ba8461cd88af20fd66a3e44055deb24f826e"
Does anyone see what is wrong with the code?
EDIT: full stacktrace with a quite empty repo (only one commit and a change on the only line in the only file):
Exception in thread "main" org.eclipse.jgit.errors.MissingObjectException: Missing blob f7891cbde46bbb6ca96065ecf1900ef6a223f679
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:149)
at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:140)
at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:276)
at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1020)
at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:950)
at org.eclipse.jgit.diff.DiffFormatter.toFileHeader(DiffFormatter.java:915)
at MainKt.main(Main.kt:17)
Not sure if this will be helpful, but I lost some time in this issue and if someone falls down this hole in the future: It appears to be some issue in the DiffFormatter class when the latest changes which you are trying to diff are not committed.
My use case was to get the changes of a single file, and to make it work, I made use of the gitDiff interface directly, like so:
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
String filename = "mafile.java";
Git.open(File("/path/toMyRepo")).diff() // calling diff command
.setPathFilter(PathFilter.create(filename)) //setting to just look for changes in specific file
.setContextLines(0) // just the changes itself, not the bordering lines
.setOutputStream(byteArrayOutputStream) // the outputstream to print the changes
.call();
System.out.println(byteArrayOutputStream.toString(StandardCharsets.UTF_8));

AmazonS3: Getting warning: S3AbortableInputStream:Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection

Here's the warning that I am getting:
S3AbortableInputStream:Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
I tried using try with resources but S3ObjectInputStream doesn't seem to close via this method.
try (S3Object s3object = s3Client.getObject(new GetObjectRequest(bucket, key));
S3ObjectInputStream s3ObjectInputStream = s3object.getObjectContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(s3ObjectInputStream, StandardCharsets.UTF_8));
){
//some code here blah blah blah
}
I also tried below code and explicitly closing but that doesn't work either:
S3Object s3object = s3Client.getObject(new GetObjectRequest(bucket, key));
S3ObjectInputStream s3ObjectInputStream = s3object.getObjectContent();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(s3ObjectInputStream, StandardCharsets.UTF_8));
){
//some code here blah blah
s3ObjectInputStream.close();
s3object.close();
}
Any help would be appreciated.
PS: I am only reading two lines of the file from S3 and the file has more data.
Got the answer via other medium. Sharing it here:
The warning indicates that you called close() without reading the whole file. This is problematic because S3 is still trying to send the data and you're leaving the connection in a sad state.
There's two options here:
Read the rest of the data from the input stream so the connection can be reused.
Call s3ObjectInputStream.abort() to close the connection without reading the data. The connection won't be reused, so you take some performance hit with the next request to re-create the connection. This may be worth it if it's going to take a long time to read the rest of the file.
Following option #1 of Chirag Sejpal's answer I used the below statement to drain the S3AbortableInputStream to ensure the connection can be reused:
com.amazonaws.util.IOUtils.drainInputStream(s3ObjectInputStream);
I ran into the same problem and the following class helped me
#Data
#AllArgsConstructor
public class S3ObjectClosable implements Closeable {
private final S3Object s3Object;
#Override
public void close() throws IOException {
s3Object.getObjectContent().abort();
s3Object.close();
}
}
and now you can use without warning
try (final var s3ObjectClosable = new S3ObjectClosable(s3Client.getObject(bucket, key))) {
//same code
}
To add an example to Chirag Sejpal's answer (elaborating on option #1), the following can be used to read the rest of the data from the input stream before closing it:
S3Object s3object = s3Client.getObject(new GetObjectRequest(bucket, key));
try (S3ObjectInputStream s3ObjectInputStream = s3object.getObjectContent()) {
try {
// Read from stream as necessary
} catch (Exception e) {
// Handle exceptions as necessary
} finally {
while (s3ObjectInputStream != null && s3ObjectInputStream.read() != -1) {
// Read the rest of the stream
}
}
// The stream will be closed automatically by the try-with-resources statement
}
I ran into the same error.
As others have pointed out, the /tmp space in lambda is limited to 512 MB.
And if the lambda context is re-used for a new invocation, then the /tmp space is already half-full.
So, when reading the S3 objects and writing all the files to the /tmp directory (as I was doing),
I ran out of disk space somewhere in between.
Lambda exited with error, but NOT all bytes from the S3ObjectInputStream were read.
So, two things one need to keep in mind:
1) If the first execution causes the problem, be stingy with your /tmp space.
We have only 512 MB
2) If the second execution causes the problem, then this could be resolved by attacking the root problem.
Its not possible to delete the /tmp folder.
So, delete all the files in the /tmp folder after the execution is finished.
In java, here is what I did, which successfully resolved the problem.
public String handleRequest(Map < String, String > keyValuePairs, Context lambdaContext) {
try {
// All work here
} catch (Exception e) {
logger.error("Error {}", e.toString());
return "Error";
} finally {
deleteAllFilesInTmpDir();
}
}
private void deleteAllFilesInTmpDir() {
Path path = java.nio.file.Paths.get(File.separator, "tmp", File.separator);
try {
if (Files.exists(path)) {
deleteDir(path.toFile());
logger.info("Successfully cleaned up the tmp directory");
}
} catch (Exception ex) {
logger.error("Unable to clean up the tmp directory");
}
}
public void deleteDir(File dir) {
File[] files = dir.listFiles();
if (files != null) {
for (final File file: files) {
deleteDir(file);
}
}
dir.delete();
}
This is my solution. I'm using spring boot 2.4.3
Create an amazon s3 client
AmazonS3 amazonS3Client = AmazonS3ClientBuilder
.standard()
.withRegion("your-region")
.withCredentials(
new AWSStaticCredentialsProvider(
new BasicAWSCredentials("your-access-key", "your-secret-access-key")))
.build();
Create an amazon transfer client.
TransferManager transferManagerClient = TransferManagerBuilder.standard()
.withS3Client(amazonS3Client)
.build();
Create a temporary file in /tmp/{your-s3-key} so that we can put the file we download in this file.
File file = new File(System.getProperty("java.io.tmpdir"), "your-s3-key");
try {
file.createNewFile(); // Create temporary file
} catch (IOException e) {
e.printStackTrace();
}
file.mkdirs(); // Create the directory of the temporary file
Then, we download the file from s3 using transfer manager client
// Note that in this line the s3 file downloaded has been transferred in to the temporary file that we created
Download download = transferManagerClient.download(
new GetObjectRequest("your-s3-bucket-name", "your-s3-key"), file);
// This line blocks the thread until the download is finished
download.waitForCompletion();
Now that the s3 file has been successfully transferred into the temporary file that we created. We can get the InputStream of the temporary file.
InputStream input = new DataInputStream(new FileInputStream(file));
Because the temporary file is not needed anymore, we just delete it.
file.delete();

How to Read a pre-built Text File in a Windows Phone Application

I've been trying to read a pre-built file with Car Maintenance tips, there's one in each line of my "Tips.txt" file. I've tried to follow around 4 or 5 different approaches but It's not working, it compiles but I get an exception. Here's what I've got:
using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
{
using (StreamReader sr = new StreamReader(store.OpenFile("Tips.txt", FileMode.Open, FileAccess.Read)))
{
string line;
while ((line = sr.ReadLine()) != null)
{
(App.Current as App).MyTips.Insert(new DoubleNode(line));
}
}
}
I'm getting this "Operation not permitted on IsolatedStorageFileStream", from the info inside the 2nd using statement. I tried with the build action of my "Tips.txt" set to resource, and content, yet I get the same result.
Thanks in advance.
Since you've added it to your project directory, you can't read it using Isolated Storage methods. There are various ways you can load the file. One way would be to set the text file's build type to Resource, then read it in as a stream:
//Replace 'MyProject' with the name of your XAP/Project
Stream txtStream = Application.GetResourceStream(new Uri("/MyProject;component/myTextFile.txt",
UriKind.Relative)).Stream;
using(StreamReader sr = new StreamReader(txtStream))
{
//your code
}

Azure storage: Uploaded files with size zero bytes

When I upload an image file to a blob, the image is uploaded apparently successfully (no errors). When I go to cloud storage studio, the file is there, but with a size of 0 (zero) bytes.
The following is the code that I am using:
// These two methods belong to the ContentService class used to upload
// files in the storage.
public void SetContent(HttpPostedFileBase file, string filename, bool overwrite)
{
CloudBlobContainer blobContainer = GetContainer();
var blob = blobContainer.GetBlobReference(filename);
if (file != null)
{
blob.Properties.ContentType = file.ContentType;
blob.UploadFromStream(file.InputStream);
}
else
{
blob.Properties.ContentType = "application/octet-stream";
blob.UploadByteArray(new byte[1]);
}
}
public string UploadFile(HttpPostedFileBase file, string uploadPath)
{
if (file.ContentLength == 0)
{
return null;
}
string filename;
int indexBar = file.FileName.LastIndexOf('\\');
if (indexBar > -1)
{
filename = DateTime.UtcNow.Ticks + file.FileName.Substring(indexBar + 1);
}
else
{
filename = DateTime.UtcNow.Ticks + file.FileName;
}
ContentService.Instance.SetContent(file, Helper.CombinePath(uploadPath, filename), true);
return filename;
}
// The above code is called by this code.
HttpPostedFileBase newFile = Request.Files["newFile"] as HttpPostedFileBase;
ContentService service = new ContentService();
blog.Image = service.UploadFile(newFile, string.Format("{0}{1}", Constants.Paths.BlogImages, blog.RowKey));
Before the image file is uploaded to the storage, the Property InputStream from the HttpPostedFileBase appears to be fine (the size of the of image corresponds to what is expected! And no exceptions are thrown).
And the really strange thing is that this works perfectly in other cases (uploading Power Points or even other images from the Worker role). The code that calls the SetContent method seems to be exactly the same and file seems to be correct since a new file with zero bytes is created at the correct location.
Does any one have any suggestion please? I debugged this code dozens of times and I cannot see the problem. Any suggestions are welcome!
Thanks
The Position property of the InputStream of the HttpPostedFileBase had the same value as the Length property (probably because I had another file previous to this one - stupid I think!).
All I had to do was to set the Position property back to 0 (zero)!
I hope this helps somebody in the future.
Thanks Fabio for bringing this up and solving your own question. I just want to add code to whatever you have said. Your suggestion worked perfectly for me.
var memoryStream = new MemoryStream();
// "upload" is the object returned by fine uploader
upload.InputStream.CopyTo(memoryStream);
memoryStream.ToArray();
// After copying the contents to stream, initialize it's position
// back to zeroth location
memoryStream.Seek(0, SeekOrigin.Begin);
And now you are ready to upload memoryStream using:
blockBlob.UploadFromStream(memoryStream);