Weird Characters when reading Blob from Container - azure-powershell

I have some text files that contain either OK or FAIL written to a storage account. If I download the files to disk, the contents display as expected in notepad,
However, if I get the file contents to a variable and Write-Host that, I get �O K
I am doing this:
$storageContext = New-AzureStorageContext $storageAccountName $storageAccountKey
$storageContainer = "monitor"
$storageBlobs = Get-AzureStorageBlob -Container $storageContainer -Context $storageContext
foreach($storageBlob in $storageBlobs) {
$blobContents = $storageBlob.ICloudBlob.DownloadText()
$blobName = $storageBlob.Name
Write-Host $blobName
Write-Host $blobContents
Write-Host
}
I thought that maybe I could tell it to use UTF8 like this:
$encoding = [System.text.Encoding]::UTF8
foreach($storageBlob in $storageBlobs) {
$blobContents = $storageBlob.ICloudBlob.DownloadText($encoding)
$blobName = $storageBlob.Name
Write-Host $blobName
Write-Host $blobContents
Write-Host
}
However, that doesn't work.
I'm now trying getting it as a ByteArray with .DownloadToByteArray but my first attempt is throwing errors.
Why are there extra characters and spaces when retrieved via Powershell but not when viewed in notepad. Am I correct in thinking this is a simple encoding issue?

Yes, it is an encoding issue. If the file is encoding with UTF8, you should use UTF8 to decode it. If it's encoding with unicode, you should use unicode to decode it.
Then in your case, I think it's using unicode. You should use Unicode to decode it, like $encoding = [System.text.Encoding]::Unicode.
I can reproduce your issue, and test it with encoding UTF8(not work), but works with Unicode.
UTF8:
Unicode:

Related

Perl6: large gzipped files read line by line

I'm trying to read a gz file line by line in Perl6, however, I'm getting blocked:
How to read gz file line by line in Perl6 however, this method, reading everything into :out uses far too much RAM to be usable except on very small files.
I don't understand how to use Perl6's Compress::Zlib to get everything line by line, although I opened an issue on their github https://github.com/retupmoca/P6-Compress-Zlib/issues/17
I'm trying Perl5's Compress::Zlib to translate this code, which works perfectly in Perl5:
use Compress::Zlib;
my $file = "data.txt.gz";
my $gz = gzopen($file, "rb") or die "Error reading $file: $gzerrno";
while ($gz->gzreadline($_) > 0) {
# Process the line read in $_
}
die "Error reading $file: $gzerrno" if $gzerrno != Z_STREAM_END ;
$gz->gzclose() ;
to something like this using Inline::Perl5 in Perl6:
use Compress::Zlib:from<Perl5>;
my $file = 'chrMT.1.vcf.gz';
my $gz = Compress::Zlib::new(gzopen($file, 'r');
while ($gz.gzreadline($_) > 0) {
print $_;
}
$gz.gzclose();
but I can't see how to translate this :(
I'm confused by Lib::Archive example https://github.com/frithnanth/perl6-Archive-Libarchive/blob/master/examples/readfile.p6 I don't see how I can get something like item 3 here
There should be something like
for $file.IO.lines(gz) -> $line { or something like that in Perl6, if it exists, I can't find it.
How can I read a large file line by line without reading everything into RAM in Perl6?
Update Now tested, which revealed an error, now fixed.
Solution #2
use Compress::Zlib;
my $file = "data.txt.gz" ;
my $handle = try open $file or die "Error reading $file: $!" ;
my $zwrap = zwrap($handle, :gzip) ;
for $zwrap.lines {
.print
}
CATCH { default { die "Error reading $file: $_" } }
$handle.close ;
I've tested this with a small gzipped text file.
I don't know much about gzip etc. but figured this out based on:
Knowing P6;
Reading Compress::Zlib's README and choosing the zwrap routine;
Looking at the module's source code, in particular the signature of the zwrap routine our sub zwrap ($thing, :$zlib, :$deflate, :$gzip);
And trial and error, mainly to guess that I needed to pass the :gzip adverb.
Please comment on whether my code works for you. I'm guessing the main thing is whether it's fast enough for the large files you have.
A failed attempt at solution #5
With solution #2 working I would have expected to be able to write just:
use Compress::Zlib ;
.print for "data.txt.gz".&zwrap(:gzip).lines ;
But that fails with:
No such method 'eof' for invocant of type 'IO::Path'
This is presumably because this module was written before the reorganization of the IO classes.
That led me to #MattOates' IO::Handle like object with .lines ? issue. I note no response and I saw no related repo at https://github.com/MattOates?tab=repositories.
I am focusing on the Inline::Perl5 solution that you tried.
For the call to $gz.gzreadline($_): it seems like gzreadline tries to return the line read from the zip file by modifying its input argument $_ (treated as an output argument, but it is not a true Perl 5 reference variable[1]), but the modified value is not returned to the Perl 6 script.
Here is a possoble workaround:
Create a wrapper module in the curent directory, e.g. ./MyZlibWrapper.pm:
package MyZlibWrapper;
use strict;
use warnings;
use Compress::Zlib ();
use Exporter qw(import);
our #EXPORT = qw(gzopen);
our $VERSION = 0.01;
sub gzopen {
my ( $fn, $mode ) = #_;
my $gz = Compress::Zlib::gzopen( $fn, $mode );
my $self = {gz => $gz};
return bless $self, __PACKAGE__;
}
sub gzreadline {
my ( $self ) = #_;
my $line = "";
my $res = $self->{gz}->gzreadline($line);
return [$res, $line];
}
sub gzclose {
my ( $self ) = #_;
$self->{gz}->gzclose();
}
1;
Then use Inline::Perl5 on this wrapper module instead of Compress::Zlib. For example ./p.p6:
use v6;
use lib:from<Perl5> '.';
use MyZlibWrapper:from<Perl5>;
my $file = 'data.txt.gz';
my $mode = 'rb';
my $gz = gzopen($file, $mode);
loop {
my ($res, $line) = $gz.gzreadline();
last if $res == 0;
print $line;
}
$gz.gzclose();
[1]
In Perl 5 you can modify an input argument that is not a reference, and the change will be reflected in the caller. This is done by modifying entries in the special #_ array variable. For example: sub quote { $_[0] = "'$_[0]'" } $str = "Hello"; quote($str) will quote $str even if $str is not passed by reference.

StreamWriter cannot call a method on a null-valued expression

First time user, looking for help with a script that's been driving me crazy.
Basically, I need to create a set number of files of an exact size (512KB, 2MB, 1GB) to test a SAN. These files need to be filled with random text so that the SAN doesn't catch the nuls and does actually allocate the blocks - that's also the reason I couldn't just use fsutils.
Now, I've been messing with the new-bigrandomfile by Verboon and tweaking it to my needs.
However I'm getting the error:
You cannot call a method on a null-valued expression.
At L:\random5.ps1:34 char:9
+ $stream.Write($longstring)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
This is the bit of code I've come up with so far; I'll add a loop at the end to copy the file I just created N times so to fill up the lun.
Set-Strictmode -Version 2.0
#temp file
$file = "c:\temp\temp.rnd"
#charset size
$charset = 64
#Block Size
$blocksize = 512
#page size
$Pagesize = 512KB
#Number of blocks in a page
$blocknum = $Pagesize / $blocksize
#Resulting/desired test file size
$filesize = 1GB
#number of pages in a file
$pagenum = $filesize / $Pagesize
# create the stream writer
$stream = System.IO.StreamWriter $file
# get a 64 element Char[]; I added the - and _ to have 64 chars
[char[]]$chars = 'azertyuiopqsdfghjklmwxcvbnAZERTYUIOPQSDFGHJKLMWXCVBN0123456789-_'
1..$Pagenum | ForEach-Object {
# get a page's worth of blocks
1..$blocknum| ForEach-Object {
# randomize all chars and...
$rndChars = $chars | Get-Random -Count $chars.Count
# ...join them in a string
$string = -join $rndChars
# repeat random string N times to get a full block string length
$longstring = $string * ($blocksize / $charset)
# write 1 block to file
$stream.Write($longstring)
# release resources by clearing string variables
Clear-Variable string, longstring
}
}
$stream.Close()
$stream.Dispose()
# release resources through garbage collection
[GC]::Collect()
$file.Close()
I've tried a gazillion variants like:
$stream = [System.IO.StreamWriter] $file
$stream = System.IO.StreamWriter $file
$stream = NewObject System.IO.StreamWriter $file
Of course, being a total noob at powershell, I've tried using quotes, brackets, provided the full path instead of the variable, etc. All (or most) seem to be valid syntax variants, according to a ton of examples I found online, but the output is still the same.
In case you have any improvement to suggest or alternative way to perform this task I'm all ears.
Edited the script above: just a couple of " for $file made the error disappear, - thanks LinuxDisciple; however, the file gets created but stays at 0 bytes and the script stuck in a loop.
Fix your instantiation of StreamWriter to any of these correct variants:
$stream = [System.IO.StreamWriter]::new($file)
$stream = [IO.StreamWriter]::new($file) # the default namespace may be omitted
$stream = New-Object System.IO.StreamWriter $file
You can specify encoding:
$stream = [IO.StreamWriter]::new(
$file,
$false, # don't append
[Text.Encoding]::ASCII
)
See StreamWriter on MSDN for available constructors and parameters.
PowerShell ISE offers autocomplete with tooltips:
type [streamw and press Ctrl-Space to autocomplete the full .NET class name
type ]:: to see the available methods and properties
type new and press Ctrl-Space to see the constructor overrides
whenever needed, put the caret at the method name and press Ctrl-Space for the tooltip
I know nothing about powershell but a few things:
Are you sure $longstring has a value before you call stream.Write()? It sounds like it's null and that's why the error. If you can somehow output the value of $longstring to the console, it would help you make sure that it has a value.
Also, troubleshoot the code with a simplified version of your code, so that you can pinpoint what's going on, for example
$file = c:\temp\temp.rnd
$stream = System.IO.StreamWriter $file
$longstring = 'whatever'
$stream.Write($longstring)

Safe late variable expansion in Powershell

I found a very old thread here that seemed to answer the question, but when I tried to implement the code I am not getting the expected variable expansion. Based on this
$Arguments = '$foo (Notepad.exe) bar'
$foo = 'Foooooo'
$InitSB = {$ExecutionContext.SessionState.Applications.Clear(); $ExecutionContext.SessionState.Scripts.Clear(); Get-Command | %{$_.Visibility = 'Private'}}
$SafeStringEvalSB = {param($str) $str}
$job = Start-Job -Init $InitSB -ScriptBlock $SafeStringEvalSB -ArgumentList $Arguments
Wait-Job $job > $null
Receive-Job $job
I would expect to get back Foooooo (Notepad.exe) bar
What have I got wrong here? And is there any difference between v2 and later versions?
I do have this working using $ExecutionContext.InvokeCommand.ExpandString(), but that allows for arbitrary code to execute as well, which I am trying to avoid. Ultimately I have a large number of different strings stored in XML, with different permutations of variables interspersed in the strings, and the actual values of those different variables are assigned at run time. And I need to push those values into the specific string of many possible wherever they occur. Because there are so many potential strings and variables I can't easily use a simple format approach. So ultimately I need the functionality of $ExecutionContext.InvokeCommand.ExpandString() but limited to actually just expanding variables, with no other code execution.
Any help greatly appreciated.
Alternatively you could just replace the text.
Param($var1,$var2,$var3)
$Content = Get-Content C:\Path\To\file.xml
$Content -replace '\$var1', "$var1" -replace '\$var2', "$var2" -replace '\$var3', "$var3"
As you have it written you won't get what you expect purely because you have single quotes around your string that you are assigning to $Arguments, meaning $foo wouldn't be treated as a variable anyway.
EDIT: Still won't work even with double quotes. Must assign value to $foo before referencing it in $Arguments.
$foo = 'Foooooo'
$Arguments = "$foo (Notepad.exe) bar"
$InitSB = {$ExecutionContext.SessionState.Applications.Clear(); $ExecutionContext.SessionState.Scripts.Clear(); Get-Command | %{$_.Visibility = 'Private'}}
$SafeStringEvalSB = {param($str) $str}
$job = Start-Job -Init $InitSB -ScriptBlock $SafeStringEvalSB -ArgumentList $Arguments
Wait-Job $job > $null
Receive-Job $job

Running Powershell script with if condition from a .NET app

I have a .net application that successfully runs Powershell commands that it pulls from text files- until I tried doing one that is more complicated and contains an if condition. The script works correctly from a PS console but in .NET I only know how to pass in a string for the script, which after reading the file, it adds extra stuff like vblf and even if I take it out, it won't work. Is this even possible?
.NET Runtime Error: Server was unable to process request. ---> The
term 'False' is not recognized as the name of a cmdlet, function,
script file, or operable program. Check the spelling of the name, or
if a path was included, verify that the path is correct and try again.
.NET code:
'Grab Powershell script from text (.ps1) file
strScript = File.ReadAllText(ScriptFileName)
'inject the arguments into the script
strScript = InsertArguments(strScript, Arguments)
'Open the runspace and create a pipeline if it's not already open
If psRunspace.RunspaceStateInfo.State = RunspaceState.BeforeOpen Then
psRunspace.Open()
End If
Dim MyPipeline As Pipeline = psRunspace.CreatePipeline()
MyPipeline.Commands.AddScript(strScript)
Dim psResults As Collection(Of PSObject) = MyPipeline.Invoke()
Powershell Script, stored in ps1 file:
new-mailbox -name $argument1 -DisplayName $argument1 -UserPrincipalName $argument2 Room -DomainController $argument5
if ($argument4 -eq "False") {
Set-CalendarProcessing $argument1 -BookingWindowInDays 400 -DeleteSubject $false -AutomateProcessing autoaccept -AllBookInPolicy $false -BookInPolicy $argument3 -DomainController $argument5
} else {
Set-CalendarProcessing $argument1 -BookingWindowInDays 400 -DeleteSubject $false -AutomateProcessing autoaccept -AllBookInPolicy $true -DomainController $argument5
}
When this script is read in, here is a substring of what gets pulled into strScript:
-Room -DomainController mcexdct1" & vbLf & "if (False -eq "False") {"
Fixed it by changing the PowerShell script from this:
if ($argument4 -eq "False") {
to this:
if ("$argument4" -eq "False") {
It works with quotation marks around it. I guess the "junk" I was seeing in the script string (like vblf) is normal- I don't work with reading from text files that much.

Unicode characters not showing in Zend_Pdf?

require_once 'Zend/Pdf.php';
$pdf = new Zend_Pdf();
$page = $pdf->newPage(Zend_Pdf_Page::SIZE_A4);
$pdf->pages[] = $page;
$page->setFont(Zend_Pdf_Font::fontWithName(Zend_Pdf_Font::FONT_HELVETICA), 10);
$page->drawText("Bogus Russian: это фигня", 100, 400, "UTF-8");
$pdfData = $pdf->render();
header("Content-Disposition: inline; filename=output.pdf");
header("Content-type: application/x-pdf");
echo $pdfData;
I can't get the Russian characters to show up! I've managed to get them to show up as:
Russian: ???????????
Russian: ÐоммÑнÐ
Russian:
and
Russian: ><
This post explains it better:
How to generate pdf files _with_ utf-8 multibyte characters using Zend Framework
Essentially the built-in fonts (eg. Zend_Pdf_Font::FONT_HELVETICA) don't contain enough info, so you need to attach a font ttf file with the pdf, and set your script to use that font.
$font = Zend_Pdf_Font::fontWithPath('/Library/Fonts/Times.ttf');
$pdfPage->setFont($font, 36);
Perhaps this will answer your question:
How to generate pdf files _with_ utf-8 multibyte characters using Zend Framework
By the looks of it the default fonts do not have all the utf-8 characters...you may have to load external TTF...