StreamWriter cannot call a method on a null-valued expression - nullpointerexception

First time user, looking for help with a script that's been driving me crazy.
Basically, I need to create a set number of files of an exact size (512KB, 2MB, 1GB) to test a SAN. These files need to be filled with random text so that the SAN doesn't catch the nuls and does actually allocate the blocks - that's also the reason I couldn't just use fsutils.
Now, I've been messing with the new-bigrandomfile by Verboon and tweaking it to my needs.
However I'm getting the error:
You cannot call a method on a null-valued expression.
At L:\random5.ps1:34 char:9
+ $stream.Write($longstring)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
This is the bit of code I've come up with so far; I'll add a loop at the end to copy the file I just created N times so to fill up the lun.
Set-Strictmode -Version 2.0
#temp file
$file = "c:\temp\temp.rnd"
#charset size
$charset = 64
#Block Size
$blocksize = 512
#page size
$Pagesize = 512KB
#Number of blocks in a page
$blocknum = $Pagesize / $blocksize
#Resulting/desired test file size
$filesize = 1GB
#number of pages in a file
$pagenum = $filesize / $Pagesize
# create the stream writer
$stream = System.IO.StreamWriter $file
# get a 64 element Char[]; I added the - and _ to have 64 chars
[char[]]$chars = 'azertyuiopqsdfghjklmwxcvbnAZERTYUIOPQSDFGHJKLMWXCVBN0123456789-_'
1..$Pagenum | ForEach-Object {
# get a page's worth of blocks
1..$blocknum| ForEach-Object {
# randomize all chars and...
$rndChars = $chars | Get-Random -Count $chars.Count
# ...join them in a string
$string = -join $rndChars
# repeat random string N times to get a full block string length
$longstring = $string * ($blocksize / $charset)
# write 1 block to file
$stream.Write($longstring)
# release resources by clearing string variables
Clear-Variable string, longstring
}
}
$stream.Close()
$stream.Dispose()
# release resources through garbage collection
[GC]::Collect()
$file.Close()
I've tried a gazillion variants like:
$stream = [System.IO.StreamWriter] $file
$stream = System.IO.StreamWriter $file
$stream = NewObject System.IO.StreamWriter $file
Of course, being a total noob at powershell, I've tried using quotes, brackets, provided the full path instead of the variable, etc. All (or most) seem to be valid syntax variants, according to a ton of examples I found online, but the output is still the same.
In case you have any improvement to suggest or alternative way to perform this task I'm all ears.
Edited the script above: just a couple of " for $file made the error disappear, - thanks LinuxDisciple; however, the file gets created but stays at 0 bytes and the script stuck in a loop.

Fix your instantiation of StreamWriter to any of these correct variants:
$stream = [System.IO.StreamWriter]::new($file)
$stream = [IO.StreamWriter]::new($file) # the default namespace may be omitted
$stream = New-Object System.IO.StreamWriter $file
You can specify encoding:
$stream = [IO.StreamWriter]::new(
$file,
$false, # don't append
[Text.Encoding]::ASCII
)
See StreamWriter on MSDN for available constructors and parameters.
PowerShell ISE offers autocomplete with tooltips:
type [streamw and press Ctrl-Space to autocomplete the full .NET class name
type ]:: to see the available methods and properties
type new and press Ctrl-Space to see the constructor overrides
whenever needed, put the caret at the method name and press Ctrl-Space for the tooltip

I know nothing about powershell but a few things:
Are you sure $longstring has a value before you call stream.Write()? It sounds like it's null and that's why the error. If you can somehow output the value of $longstring to the console, it would help you make sure that it has a value.
Also, troubleshoot the code with a simplified version of your code, so that you can pinpoint what's going on, for example
$file = c:\temp\temp.rnd
$stream = System.IO.StreamWriter $file
$longstring = 'whatever'
$stream.Write($longstring)

Related

Perl6: large gzipped files read line by line

I'm trying to read a gz file line by line in Perl6, however, I'm getting blocked:
How to read gz file line by line in Perl6 however, this method, reading everything into :out uses far too much RAM to be usable except on very small files.
I don't understand how to use Perl6's Compress::Zlib to get everything line by line, although I opened an issue on their github https://github.com/retupmoca/P6-Compress-Zlib/issues/17
I'm trying Perl5's Compress::Zlib to translate this code, which works perfectly in Perl5:
use Compress::Zlib;
my $file = "data.txt.gz";
my $gz = gzopen($file, "rb") or die "Error reading $file: $gzerrno";
while ($gz->gzreadline($_) > 0) {
# Process the line read in $_
}
die "Error reading $file: $gzerrno" if $gzerrno != Z_STREAM_END ;
$gz->gzclose() ;
to something like this using Inline::Perl5 in Perl6:
use Compress::Zlib:from<Perl5>;
my $file = 'chrMT.1.vcf.gz';
my $gz = Compress::Zlib::new(gzopen($file, 'r');
while ($gz.gzreadline($_) > 0) {
print $_;
}
$gz.gzclose();
but I can't see how to translate this :(
I'm confused by Lib::Archive example https://github.com/frithnanth/perl6-Archive-Libarchive/blob/master/examples/readfile.p6 I don't see how I can get something like item 3 here
There should be something like
for $file.IO.lines(gz) -> $line { or something like that in Perl6, if it exists, I can't find it.
How can I read a large file line by line without reading everything into RAM in Perl6?
Update Now tested, which revealed an error, now fixed.
Solution #2
use Compress::Zlib;
my $file = "data.txt.gz" ;
my $handle = try open $file or die "Error reading $file: $!" ;
my $zwrap = zwrap($handle, :gzip) ;
for $zwrap.lines {
.print
}
CATCH { default { die "Error reading $file: $_" } }
$handle.close ;
I've tested this with a small gzipped text file.
I don't know much about gzip etc. but figured this out based on:
Knowing P6;
Reading Compress::Zlib's README and choosing the zwrap routine;
Looking at the module's source code, in particular the signature of the zwrap routine our sub zwrap ($thing, :$zlib, :$deflate, :$gzip);
And trial and error, mainly to guess that I needed to pass the :gzip adverb.
Please comment on whether my code works for you. I'm guessing the main thing is whether it's fast enough for the large files you have.
A failed attempt at solution #5
With solution #2 working I would have expected to be able to write just:
use Compress::Zlib ;
.print for "data.txt.gz".&zwrap(:gzip).lines ;
But that fails with:
No such method 'eof' for invocant of type 'IO::Path'
This is presumably because this module was written before the reorganization of the IO classes.
That led me to #MattOates' IO::Handle like object with .lines ? issue. I note no response and I saw no related repo at https://github.com/MattOates?tab=repositories.
I am focusing on the Inline::Perl5 solution that you tried.
For the call to $gz.gzreadline($_): it seems like gzreadline tries to return the line read from the zip file by modifying its input argument $_ (treated as an output argument, but it is not a true Perl 5 reference variable[1]), but the modified value is not returned to the Perl 6 script.
Here is a possoble workaround:
Create a wrapper module in the curent directory, e.g. ./MyZlibWrapper.pm:
package MyZlibWrapper;
use strict;
use warnings;
use Compress::Zlib ();
use Exporter qw(import);
our #EXPORT = qw(gzopen);
our $VERSION = 0.01;
sub gzopen {
my ( $fn, $mode ) = #_;
my $gz = Compress::Zlib::gzopen( $fn, $mode );
my $self = {gz => $gz};
return bless $self, __PACKAGE__;
}
sub gzreadline {
my ( $self ) = #_;
my $line = "";
my $res = $self->{gz}->gzreadline($line);
return [$res, $line];
}
sub gzclose {
my ( $self ) = #_;
$self->{gz}->gzclose();
}
1;
Then use Inline::Perl5 on this wrapper module instead of Compress::Zlib. For example ./p.p6:
use v6;
use lib:from<Perl5> '.';
use MyZlibWrapper:from<Perl5>;
my $file = 'data.txt.gz';
my $mode = 'rb';
my $gz = gzopen($file, $mode);
loop {
my ($res, $line) = $gz.gzreadline();
last if $res == 0;
print $line;
}
$gz.gzclose();
[1]
In Perl 5 you can modify an input argument that is not a reference, and the change will be reflected in the caller. This is done by modifying entries in the special #_ array variable. For example: sub quote { $_[0] = "'$_[0]'" } $str = "Hello"; quote($str) will quote $str even if $str is not passed by reference.

Error: Input array is longer than number of columns in this table powershell

I am trying to load 160gb csv file to sql and I am using powershell script I got from Github and I get this error
IException calling "Add" with "1" argument(s): "Input array is longer than the number of columns in this table."
At C:\b.ps1:54 char:26
+ [void]$datatable.Rows.Add <<<< ($line.Split($delimiter))
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
So I checked the same code with small 3 line csv and all of the columns match and also have header in first row and there are no extra delimiters not sure why I am getting this error.
The code is below
<# 8-faster-runspaces.ps1 #>
# Set CSV attributes
$csv = "M:\d\s.txt"
$delimiter = "`t"
# Set connstring
$connstring = "Data Source=.;Integrated Security=true;Initial Catalog=PresentationOptimized;PACKET SIZE=32767;"
# Set batchsize to 2000
$batchsize = 2000
# Create the datatable
$datatable = New-Object System.Data.DataTable
# Add generic columns
$columns = (Get-Content $csv -First 1).Split($delimiter)
foreach ($column in $columns) {
[void]$datatable.Columns.Add()
}
# Setup runspace pool and the scriptblock that runs inside each runspace
$pool = [RunspaceFactory]::CreateRunspacePool(1,5)
$pool.ApartmentState = "MTA"
$pool.Open()
$runspaces = #()
# Setup scriptblock. This is the workhorse. Think of it as a function.
$scriptblock = {
Param (
[string]$connstring,
[object]$dtbatch,
[int]$batchsize
)
$bulkcopy = New-Object Data.SqlClient.SqlBulkCopy($connstring,"TableLock")
$bulkcopy.DestinationTableName = "abc"
$bulkcopy.BatchSize = $batchsize
$bulkcopy.WriteToServer($dtbatch)
$bulkcopy.Close()
$dtbatch.Clear()
$bulkcopy.Dispose()
$dtbatch.Dispose()
}
# Start timer
$time = [System.Diagnostics.Stopwatch]::StartNew()
# Open the text file from disk and process.
$reader = New-Object System.IO.StreamReader($csv)
Write-Output "Starting insert.."
while ((($line = $reader.ReadLine()) -ne $null))
{
[void]$datatable.Rows.Add($line.Split($delimiter))
if ($datatable.rows.count % $batchsize -eq 0)
{
$runspace = [PowerShell]::Create()
[void]$runspace.AddScript($scriptblock)
[void]$runspace.AddArgument($connstring)
[void]$runspace.AddArgument($datatable) # <-- Send datatable
[void]$runspace.AddArgument($batchsize)
$runspace.RunspacePool = $pool
$runspaces += [PSCustomObject]#{ Pipe = $runspace; Status = $runspace.BeginInvoke() }
# Overwrite object with a shell of itself
$datatable = $datatable.Clone() # <-- Create new datatable object
}
}
# Close the file
$reader.Close()
# Wait for runspaces to complete
while ($runspaces.Status.IsCompleted -notcontains $true) {}
# End timer
$secs = $time.Elapsed.TotalSeconds
# Cleanup runspaces
foreach ($runspace in $runspaces ) {
[void]$runspace.Pipe.EndInvoke($runspace.Status) # EndInvoke method retrieves the results of the asynchronous call
$runspace.Pipe.Dispose()
}
# Cleanup runspace pool
$pool.Close()
$pool.Dispose()
# Cleanup SQL Connections
[System.Data.SqlClient.SqlConnection]::ClearAllPools()
# Done! Format output then display
$totalrows = 1000000
$rs = "{0:N0}" -f [int]($totalrows / $secs)
$rm = "{0:N0}" -f [int]($totalrows / $secs * 60)
$mill = "{0:N0}" -f $totalrows
Write-Output "$mill rows imported in $([math]::round($secs,2)) seconds ($rs rows/sec and $rm rows/min)"
Working with a 160 GB input file is going to be a pain. You can't really load it into any kind of editor - or at least you don't really analyze such a data mass without some serious automation.
As per the comments, it seems that the data has some quality issues. In order to find the offending data, you could try binary searching. This approach shrinks the data fast. Like so,
1) Split the file in about two equal chunks.
2) Try and load first chunk.
3) If successful, process the second chunk. If not, see 6).
4) Try and load second chunk.
5) If successful, the files are valid, but you got another a data quality issue. Start looking into other causes. If not, see 6).
6) If either load failed, start from the beginning and use the failed file as the input file.
7) Repeat until you narrow down the offending row(s).
Another a method would be using an ETL tool like SSIS. Configure the package to redirect invalid rows into an error log to see what data is not working properly.

PowerPoint to PDF: Minimum size option

I was attempting to convert PowerPoint files to PDF using PowerShell and was able to do so. However, I am trying to take this one step further and select the 'Minimum Size (publishing online)' option through the script.
Is there a property that needs to be set for this to happen? I'm guessing it is the $ppQualityStandard variable but not exactly sure.
EDIT: This is what I am using currently:
function ppt_to_pdf ($folderpath, $pptname) {
Add-Type -AssemblyName office
$ppFormatPDF = 2
$ppQualityStandard = 0
$p = New-Object -ComObject PowerPoint.Application
$p.Visible = [Microsoft.Office.Core.MsoTriState]::msoTrue
$ppt = $p.Presentations.Open("$folderpath\$pptname")
$ppt.SaveCopyAs("$folderpath\$pptname", 32)
$ppt.Close()
$p.Quit()
$p = $null
[gc]::collect()
[gc]::WaitForPendingFinalizers()
}
I suspect you need to use .ExportAsFixedFormat rather than .SaveCopyAs.
It takes, among other parameters, Intent as type ppFixedFormatIntent, which can be either:
ppFixedFormatIntentScreen (=1)
or
ppFixedFormatIntentPrint (=2)
There's a host of other parms. To learn more, start PPT, go into the VBA IDE and press F2 for the Object Browser and search for ExportAsFixedFormat

f.open() issue resulting in Unbound error for f.close()

I don't quite have an answer but I'm narrowing it down. Somehow I'm mixing/confusing types, I believe, between what is provided by commands like 'os.path' and type str().
As I've made the assignment of the logfile(s) globally, even though I can print it in the function, when the variable is used in fout = open(... it's actually a null that's being referenced, i.e. open() doesn't like/can't use the type it finds.
The error:
UnboundLocalError: local variable 'fout' referenced before assignment
I am simply writing a log of dot files (left on USB drives by OSX) for deletion, but the try/except is now falling over. First the original version.
working code:
logFile = "/Users/dee/Desktop/dotFile_names.txt"
try:
fout = open(logFile, 'w')
for line in dotFile_names:
fout.write(line)
except IOError as e:
print ("Error : %s not found." % fout)
finally:
fout.close()
Attempting better practice, I sought to put the log file specs and path as variables so they can be modified if need be - I hope to make it cross platform workable. these variables are at the head of the program, i.e. not in main(), but I pass them in and print() statements have shown me they are successfully being referenced. i.e. I get this printed:
/Users/dee/Desktop/dotFile_names.txt
Despite this the error I get is:
UnboundLocalError: local variable 'fout' referenced before assignment -
error points at the "fout.close()" line
Error producing code
logFilespec = "dotFile_names.txt"
fullLogFileSpec = []
userDesktop = os.path.join(os.path.expanduser('~'), 'Desktop')
fullLogFilespec = os.path.join(userDesktop, logFilespec)
try:
print "opening " + fullLogFilespec
fout = open(fullLogFileSpec, 'w')
for line in dotFile_names:
print "..", # are we executing this line..?
fout.write(line)
except IOError as e:
print ("Error : %s not found." % fout)
finally:
print "\nclosing " + fullLogFilespec
fout.close()
I've found that if I modify this line by converting to a string
fout = open(fullLogFileSpec, 'w')
fout = open(str(fullLogFileSpec), 'w')
the error goes away, BUT NO file is created on the Desktop!
At the very least I guess that I am passing something unrecognisable to fout = open() but it is not being caught by the except. Then when I pass something that does seem to allow fout =open() to work it seems to be a ghost?
So I figure I am lost between a String and whatever kind of reference/pointer os.path.expanduser() gives me.
I'm sure it's insanely simple. Before adding the str() code I also checked all indentation, removing them all and adding back using the editor indent hotkeys, just in case that was affecting things somehow.
OK, it looks like I was wearing my dumb glasses, I think declaring
fullLogFileSpec = []
as a list instead of a string was my error.
Similar as it is, having re-written it without that list declaration this code is working fine:
logfile_directory = os.path.join(os.path.expanduser('~'),'Desktop')
log_bf_file_spec = 'ItemsFoundByFolder_' + Deez_1.current_datetime() + '.txt'
log_by_folder = os.path.join(logfile_directory, log_bf_file_spec)
the function later calls, with no error:
fout_by_folder = open(log_by_folder, 'w')

Upload a file to Sharepoint 2010 with powershell 2.0

I'm struggling since a couple of days to upload files to Sharepoint 2010 with powershell.
I'm on a win7 machine with powershell v2 trying to upload to a SP 2010 site.
I'm having 2 major issues
$Context.web value is always empty even after Executequery() and no
error is shown. My $Context variable gets the server version (14.x.x.x.x) but nothing more
$Context.Load($variable) which always returns the error Cannot find an overload for "Load" and the argument count: "1".
I copied Sharepoint DLLs to my Win7 machine and I import the reference to my script.
The below script is a mix of many parts I took from the net.
I'v already tried unsuccessfully to add an overload on the clientcontext defining Load method without Type parameter suggested in the following post
http://soerennielsen.wordpress.com/2013/08/25/use-csom-from-powershell/
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client")
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime")
$site = "https://Root-of-my-site"
$listname = "My-folder"
$context = New-Object Microsoft.SharePoint.Client.ClientContext($site)
[Microsoft.SharePoint.Client.Web]$web = $context.Web
[Microsoft.SharePoint.Client.List]$list = $web.Lists.GetByTitle($listName)
$Folder = "C:\temp\Certificates"
$List = $Context.Web.Lists.GetByTitle($listname)
Foreach ($File in (dir $Folder))
{
$FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
$FileCreationInfo.Overwrite = $true
$FileCreationInfo.Content = get-content -encoding byte -path $File.Fullname
$FileCreationInfo.URL = $File
$Upload = $List.RootFolder.Files.Add($FileCreationInfo)
$Context.Load($Upload)
$Context.ExecuteQuery()
}
The error is
Cannot find an overload for "Load" and the argument count: "1".
At C:\temp\uploadCertToSharepoint.ps1:48 char:14
+ $Context.Load <<<< ($Upload)
+ CategoryInfo : NotSpecified: (:) [], MethodException
+ FullyQualifiedErrorId : MethodCountCouldNotFindBest
Can someone please help me sorting this issue?
I'll need to upload around 400 files with ad-hoc fields to a sharepoint site in a couple of weeks and at the moment I'm completely stuck. Running the script server side is unfortunately not possible.
Thanks,
Marco
This error occurs since ClientRuntimeContext.Load is a Generics Method:
public void Load<T>(
T clientObject,
params Expression<Func<T, Object>>[] retrievals
)
where T : ClientObject
and Generics methods are not supported natively in PowerShell (V1, V2) AFAIK.
The workaround is to invoke a generic methods using MethodInfo.MakeGenericMethod method as described in article Invoking Generic Methods on Non-Generic Classes in PowerShell
In case of ClientRuntimeContext.Load method, the following PS function could be used:
Function Invoke-LoadMethod() {
param(
$clientObjectInstance = $(throw “Please provide an Client Object instance on which to invoke the generic method”)
)
$ctx = $clientObjectInstance.Context
$load = [Microsoft.SharePoint.Client.ClientContext].GetMethod("Load")
$type = $clientObjectInstance.GetType()
$clientObjectLoad = $load.MakeGenericMethod($type)
$clientObjectLoad.Invoke($ctx,#($clientObjectInstance,$null))
}
Then, in your example the line:
$Context.Load($Upload)
could be replaced with this one:
Invoke-LoadMethod -clientObjectInstance $Upload
References
Invoking Generic Methods on Non-Generic Classes in PowerShell
Some tips and tricks of using SharePoint Client Object Model in
PowerShell. Part 1
It throws the error because in powershell 2.0 you cannot call generic method directly.
You need to create closed method using MakeGenericMethod. Try to use code below.
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client")
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime")
$site = "http://server"
$listname = "listName"
$Folder = "C:\PS\Test"
$context = New-Object Microsoft.SharePoint.Client.ClientContext($site)
[Microsoft.SharePoint.Client.Web]$web = $context.Web
[Microsoft.SharePoint.Client.List]$list = $web.Lists.GetByTitle($listName)
$method = $Context.GetType().GetMethod("Load")
$closedMethod = $method.MakeGenericMethod([Microsoft.SharePoint.Client.File])
Foreach ($File in (dir $Folder))
{
$FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
$FileCreationInfo.Overwrite = $true
$FileCreationInfo.Content = (get-content -encoding byte -path $File.Fullname)
$FileCreationInfo.URL = $File
$Upload = $List.RootFolder.Files.Add($FileCreationInfo)
$closedMethod.Invoke($Context, #($Upload, $null) )
$Context.ExecuteQuery()
}