Remove old files from Azure Data Lake store using PowerShell Runbook - azure-powershell

Do someone know how to delete every file which resides more than 90 days in a directory inside data lake store and sub directories?

This can be achieved using below code. Please check and confirm if your query is solved or not.
$PurgeDays = -90
$olddate = (Get-Date).AddDays($PurgeDays)
$year = (Get-Date).AddDays($PurgeDays).Year
$month = (Get-Date).AddDays($PurgeDays).Month
$day = (Get-Date).AddDays($PurgeDays).Day
Write-Output "Files older than $olddate are getting deleted..."
$path = "/MyFolder/$year/$month/$day"
$fileslist = Get-AzureRmDataLakeStoreChildItem -AccountName "MyADLSAccount" -Path $path
if($fileslist)
{
Remove-AzureRmDataLakeStoreItem -AccountName "MyADLSAccount" -Paths $path -Force -Recurse $true
Write-Output "Deleted the folder $path successfully"
}
else
{
Write-Output "Could not purge the folder $fileslist. Either it is not old or it is not existing."
}

Related

How to bulk check in all checked out documents in SharePoint 2010

I have SharePoint 2010 environment. Many documents are checked out by other users.
I need to bulk check-in those documents. Can anyone suggest to me a reliable way for the same?
Thanks
You could try below PowerShell.
Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue
#Variables
$WebURL="http://domain/sites/emilytest"
$LibraryURL="/Shared%20Documents" #Relative URL of a folder or library
#Get Objects
$Web = Get-SPWeb $WebURL
$Folder = $web.GetFolder($LibraryURL)
#Function to find all checked out files in a SharePoint library
Function CheckIn-CheckedOutFiles($Folder)
{
$Folder.Files | Where { $_.CheckOutStatus -ne "None" } | ForEach-Object {
write-host ($_.Name,$_.URL,$_.CheckedOutBy)
#To Check in
$_.Checkin("Checked in by Administrator")
}
#Process all sub folders
$Folder.SubFolders | ForEach-Object {
CheckIn-CheckedOutFiles $_
}
}
#Call the function to find checkedout files
CheckIn-CheckedOutFiles $Folder

Need script to delete the old backup file from storage blob , the script need to run from managed instance

Can anyone help me to get the script to remove the backup file older than 45 days from storage Blob container.
Actually i want to run the script in Managed instance ( PAAS) through SQL server agent job.
Please help me on this.
According this post.
I think you can modifier the PS script. Change $FromDate = ((Get-Date).AddDays(-6)).Date to $FromDate = ((Get-Date).AddDays(-45)).Date . The script is as follows:
#PowerShell Script to delete System Databases .bak files from Azure Block Blob
#-eq = equals
#-ne = not equals
#-lt = less than
#-gt = greater than
#-le = less than or equals
#-ge = greater than or equals
#Set All Variables
$YesterdayDate = ((Get-Date).AddDays(-1)).Date #Get Yesterday date
$FromDate = ((Get-Date).AddDays(-45)).Date #Get 6 Days back date from Today
$BlobType = "Pageblob"
$bacs = Get-ChildItem $location -Filter *.bak
$container="dwhdatabasesbackup"
$StorageAccountName="analytics"
$StorageAccountKey="xxxxxx"
$context = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
$filelist = Get-AzureStorageBlob -Container $container -Context $context
#Foreach Loop With a Condition $_.LastModified.Date -eq $YesterdayDate to make Sure there was a File Uploaded Yesterday
foreach ($filein$filelist | Where-Object {$_.LastModified.Date -eq $YesterdayDate -and $_.BlobType -eq $BlobType -and ($_.Name -like "*.bak")})
{
$Yesterdayfile = $file.Name
if ($Yesterdayfile -ne $null)
{
$FileFullLength = $Yesterdayfile.Length
$FileNameWithoutDatePart = $Yesterdayfile.SubString(0, $FileFullLength-30)
Write-Output ("File Name Without Date Part: " +$FileNameWithoutDatePart)
#Foreach Loop With a Condition $_.LastModified.Date -lt $FromDate to Remove Files those are 5 Days Old
foreach ($filein$filelist | Where-Object {$_.LastModified.Date -lt $FromDate -and $_.BlobType -eq $BlobType -and ($_.Name -like "$FileNameWithoutDatePart*.bak")})
{
$removefile = $file.Name
$RemoveFileFullLength = $removefile.Length
$ModifiedDate = $file.LastModified.Date
if (($removefile -ne $null) -and ($RemoveFileFullLength -eq $FileFullLength))
{
Write-Output ("Remove File Name: ("+$removefile +") as Modified Date: ("+ $ModifiedDate +") of File is Older Than Date: ("+ $FromDate + ")")
}
}
}
}

Disable Downloading Cached FTP File

I've got a PowerShell script that I call from VBA using Excel. The script uses WinSCP to download some datetime-named FTP and SFTP files and saves them with a static filename, overwriting the old file, on a network drive location.
The script works on first run, but after that it loads the same cached version of the file. The workaround is to change the cache settings in IE to check for newer versions of stored webpages 'every time I visit the webpage'.
The macro is used by several people and is accessed using a variety of computers. Is there a way around this that I can incorporate in my code, either in VBA or PS so they don't have to remember to go into IE to change their settings?
Script is called from VBA:
Call Shell("powershell -executionpolicy bypass & ""H:\FTP\FTP.ps1""", vbHide)
Script:
try
{
# Load WinSCP .NET assembly
Add-Type -Path "C:\Program Files (x86)\WinSCP\WinSCPnet.dll"
$localPath = "H:\Worksheets\FTP"
$remotePath = "/outgoing/data/LatestData/"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions
$sessionOptions.Protocol = [WinSCP.Protocol]::ftp
$sessionOptions.HostName =
$sessionOptions.UserName =
$sessionOptions.Password =
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Get list of files in the directory
$directoryInfo = $session.ListDirectory($remotePath)
# Select the most recent file
$latest = $directoryInfo.Files |
Where-Object { -Not $_.IsDirectory} |
Where-Object {
[System.IO.Path]::GetExtension($_.Name) -eq ".nc1" -or
[System.IO.Path]::GetExtension($_.Name) -eq ".ky1" -or
[System.IO.Path]::GetExtension($_.Name) -like ".tn*" }
Group-Object { [System.IO.Path]::GetExtension($_.Name) } |
ForEach-Object{
$_.Group | Sort-Object LastWriteTime -Descending | Select -First 1
}
$extension = [System.IO.Path]::GetExtension($latest.Name)
"GetExtension('{0}') returns '{1}'" -f $fileName, $extension
if ($latest -eq $Null)
{
Write-Host "No file found"
exit 1
}
$latest | ForEach-Object{
$extension = ([System.IO.Path]::GetExtension($_.Name)).Trim(".")
$session.GetFiles($session.EscapeFileMask($remotePath + $_.Name), "$localPath\$extension.txt" ).Check()
}
$stamp = $(Get-Date -f "yyyy-MM-dd-HHmm")
$filename = $stamp.subString(0,$stamp.length-6)
$session.GetFiles(
($remotePath + $fileName),
($localPath + $fileName + "." + $stamp)).Check()
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch [Exception]
{
Write-Host $_.Exception.Message
exit 1
}

Oracle Sql and Powershell : Execute query, print/output results

I need a way to execute a SQL (by importing a .SQL script) on a remote Oracle DB using PowerShell. In addition to this I am also trying to output the results in an .xls format in a desired folder location. To add to the fun, I would also want to run this task on an automatic schedule. Please help !
I have gotten so far :
[System.Reflection.Assembly]::LoadWithPartialName ("System.Data.OracleClient") | Out-Null
$connection = "my TNS entry"
$queryString = "my SQL query"
$command = new-Object System.Data.OracleClient.OracleCommand($queryString, $connection)
$connection.Open()
$reader = $command.ExecuteReader()
$tempArr = #()
#read all rows into a hash table
while ($reader.Read())
{
$row = #{}
for ($i = 0; $i -lt $reader.FieldCount; $i++)
{
$row[$reader.GetName($i)] = $reader.GetValue($i)
}
#convert hashtable into an array of PSObjects
$tempArr+= new-object psobject -property $row
}
$connection.Close()
write-host "Conn State--> " $connection.State
$tmpArr | Export-Csv "my File Path" -NoTypeInformation
$Error[0] | fl -Force
The easiest way is to drive sqlplus.exe via powershell. To execute the sql and get the output you do this:
$result = sqlplus.exe #file.sql [credentials/server]
#parse result into CSV here which can be loaded into excel
You can schedule this script with something like:
schtasks.exe /create /TN sqlplus /TR "Powershell -File script.ps1" /ST 10 ...
For this you need to have sqlplus installed (it comes with oracle express and you could install it without it). This obviously introduces dependency that is not needed but sqlplus could be used to examine the database and do any kind of thing which might be good thing to have around.

Variable won't store data in powershell

For some reason I can't get this variable ($SelectedDrive) to store data in it using the Get-Content cmdlet.
I have created a function as shown below:
#Allow user to select which drive to backup to and checks that the user hasn't canceled out of the Drive Selection GUI
Function SelectDrive{
ShowUSBs | Out-File $locBackupFolder\RemovableStorage.txt
Get-Content $locBackupFolder\RemovableStorage.txt | Out-GridView -OutputMode Single -Title "Choose USB Storage Device" | Out-File $locBackupFolder\Drive.txt -Encoding default
$SelectedDrive = Get-Content $locBackupFolder\Drive.txt
IF ($SelectedDrive -eq $null) {
WarningCancel
RemoveBackupDrive
Exit
} ELSE {
$BackupDrive = $SelectedDrive.Substring(0,2)
}
}
When I run the script in Powershell ISE using the F5 button it runs through, and then fails to enter the data.
I go back and check it over and can't see any issue, as it works if I run it piece by piece using Run Selection.
The whole script is below:
######### DEFINE FUNCTIONS #########
$locBackupFolder = "C:\Backup"
#Check Backup Drive
IF (!(Test-Path $locBackupFolder)) {
#C:\Backup does NOT exist
New-Item $locBackupFolder -ItemType Directory
}
#Inform user only D:\Username will be backed up
function WarningDialog(
$MessageWarning = "!!!WARNING!!!
YOU MAY LOSE DATA IF YOU DO NOT ADHERE TO THIS MESSAGE
Only D:\$env:USERNAME will be backed up, please ensure any Data that is sitting in the root of D:\ is moved to D:\$env:USERNAME
FYI: Your desktop data is safe and will be backed up, you need only worry about data in the root of D:\ or anything you have stored in
C:\ Drive.
FURTHER MORE Outlook will shutdown by itself through this process, please do not open it up again until everything is finished.
If you have data to move, please click Cancel now, otherwise please press OK to continue the backup procedure.
For any help, please see your IT Technicians, or call
off-site.",
$WindowTitleWarning = "Backup your data from your laptop",
[System.Windows.Forms.MessageBoxButtons]$ButtonsWarning = [System.Windows.Forms.MessageBoxButtons]::OKCancel,
[System.Windows.Forms.MessageBoxIcon]$IconWarning = [System.Windows.Forms.MessageBoxIcon]::Stop
)
{
Add-Type -AssemblyName System.Windows.Forms
return [System.Windows.Forms.MessageBox]::Show($MessageWarning, $WindowTitleWarning, $ButtonsWarning, $IconWarning)
}
#Checks to see if logged on user has a D:\USERNAME directory and if not, informs them to see IT for a custom backup.
Function TestUserDrive {
IF (!(Test-Path "D:\$env:USERNAME")){
#D:\USERNAME does NOT exist
}
}
#Displays an instruction/how-to of how to move data from the root of D:\ to the users D:\USERNAME folder
function WarningCancel(
$MessageCancel = "INSTRUCTIONS:
You have chosen to cancel the backup script, if this is due to requiring data to be moved to inside your D:\$env:USERNAME folder, please do the following.
1. Open My Computer
2. Double click on Data (D:) to open your D:\ Drive
3. Move or Copy anything from this directory that you wish to keep, into the directory called $env:USERNAME\My Documents
4. Once this has been completed, re-run this script to commence the backup procedure again
Easy as that!
For any help, please see your IT Technicians, or call
off-site",
$WindowTitleCancel = "Backup your data from your laptop",
[System.Windows.Forms.MessageBoxButtons]$ButtonsCancel = [System.Windows.Forms.MessageBoxButtons]::OK,
[System.Windows.Forms.MessageBoxIcon]$IconCancel = [System.Windows.Forms.MessageBoxIcon]::Information
)
{
Add-Type -AssemblyName System.Windows.Forms
return [System.Windows.Forms.MessageBox]::Show($MessageCancel, $WindowTitleCancel, $ButtonsCancel, $IconCancel)
}
#Informs the user to select the device they would like to backup to when the selection box is displayed
function SelectDevicePrompt(
$MessageSelect = "On the next screen please specify the device you would like to backup your data to.
The devices you currently have plugged in will show, please select your chosen device, and then click the OK button at the bottom right of the window.",
$WindowTitleSelect = "Backup your data from your laptop",
[System.Windows.Forms.MessageBoxButtons]$ButtonsSelect = [System.Windows.Forms.MessageBoxButtons]::OK,
[System.Windows.Forms.MessageBoxIcon]$IconSelect = [System.Windows.Forms.MessageBoxIcon]::Hand
)
{
Add-Type -AssemblyName System.Windows.Forms
return [System.Windows.Forms.MessageBox]::Show($MessageSelect, $WindowTitleSelect, $ButtonsSelect, $IconSelect)
}
#Displays a list of all removable storage devices volume names and their allocated drive letter
Function ShowUSBs{
$USBInfo = gwmi win32_diskdrive | ?{$_.interfacetype -eq "USB"} | %{gwmi -Query "ASSOCIATORS OF {Win32_DiskDrive.DeviceID=`"$($_.DeviceID.replace('\','\\'))`"} WHERE AssocClass = Win32_DiskDriveToDiskPartition"} | %{gwmi -Query "ASSOCIATORS OF {Win32_DiskPartition.DeviceID=`"$($_.DeviceID)`"} WHERE AssocClass = Win32_LogicalDiskToPartition"}
$USBInfo | Format-Table `
#{Expression={$_.DeviceID};Label="Drive Letter";Width=25}, `
#{Expression={$_.VolumeName};Label="USB Name";Width=20}
}
#Allow user to select which drive to backup to and checks that the user hasn't canceled out of the Drive Selection GUI
Function SelectDrive{
ShowUSBs | Out-File $locBackupFolder\RemovableStorage.txt
Get-Content $locBackupFolder\RemovableStorage.txt | Out-GridView -OutputMode Single -Title "Choose USB Storage Device" | Out-File $locBackupFolder\Drive.txt -Encoding default
$SelectedDrive = Get-Content $locBackupFolder\Drive.txt
IF ($SelectedDrive -eq $null) {
WarningCancel
RemoveBackupDrive
Exit
} ELSE {
$BackupDrive = $SelectedDrive.Substring(0,2)
}
}
$BackupDrive
#Imports list of active processes and looks for outlook process then kills it if found
function KillOutlook{
$processactive = Get-Process
IF($processactive.ProcessName -contains "Outlook") {
Stop-Process -Name Outlook
Start-Sleep 1
$OLcheckagain = Get-Process
IF($OLcheckagain.Processname -contains "Outlook") {
Write-Host "Outlook failed to close"
}
} Else {
Write-Host "Outlook is closed"
}
}
#Find the pst files attached to outlook and output the values to C:\Backup\PST.txt
function FindPSTs{
$outlook = New-Object -comObject Outlook.Application
$pstloc = $outlook.Session.Stores | where { ($_.FilePath -like '*.PST') }
$pstloc.FilePath | out-file -FilePath "$locBackupFolder\PST.txt" -Encoding Default
}
#Removes C:\Backup Directory
Function RemoveBackupDrive {
IF (Test-Path $locBackupFolder){
Remove-Item $locBackupFolder -Force -Recurse
}
}
#Copy data from D:\USERNAME to BackupDrive
Function CopyData {
IF (!(Test-Path $BackupDrive)) {
robocopy D:\$env:USERNAME $BackupDrive /MIR /COPYALL /r:03 /w:5 /MT:9
} ELSE {
robocopy D:\$env:USERNAME $BackupDrive /
}
}
#Copy PST files explicitly to BackupDrive\AppData\Roaming\Email
Function CopyPST {
KillOutlook
Start-Sleep 1
IF (!(Test-Path $BackupDrive\AppData\Roaming\Email)) {
New-Item $BackupDrive\AppData\Roaming\Email -ItemType Directory
}
Get-Content $locBackupFolder\PST.txt | ForEach-Object {
IF (Test-Path $_){
Copy-Item $_ "$BackupDrive\AppData\Roaming\Email"
}
}
}
######### START SCRIPT #########
#Display warning to inform user that only D:\USERNAME will be backed up
$WarningAccept = WarningDialog
#If cancel is selected from WarningDialog, then display WarningCancel message pop-up giving instructions to backup data from D:\ to D:\USERNAME
IF ($WarningAccept -eq "Cancel") {
WarningCancel
RemoveBackupDrive
Exit
}
#Prompts user to select Device to backup to
SelectDevicePrompt
#Shows the selection page for the user to select the device
SelectDrive
#Find the pst files attached to outlook and output to C:\Backup\PST.txt
FindPSTs
#Inform user where their data will be backed up to
Write-Host "Your data will be backed up to $BackupDrive"
#If Outlook is found, stop its process, otherwise continue
KillOutlook
#Running backup of everything in D:\ drive using robocopy
#If this is the first time copying all data /MIR will be used, otherwise if the directory DRIVE:\USERNAME\Backup exists
#robocopy will not /MIR and will only update newly added data.
CopyData
#Copy PST files specifically
#CopyPST
If I can be of any further assistance, please let me know.
I think your problem is simply that the SelectDrive function doesn't return anything. $backupDrive is local to the function, so you cannot refer to it from outside the function.
Also converting everything to text is going to introduce other bugs, e.g. you can select any line from the grid view including your headers and the blank lines top and bottom. Some better code might be:
Function GetUSBs{
$USBInfo = gwmi win32_diskdrive |
?{$_.interfacetype -eq "USB"} |
%{gwmi -Query "ASSOCIATORS OF {Win32_DiskDrive.DeviceID=`"$($_.DeviceID.replace('\','\\'))`"} WHERE AssocClass = Win32_DiskDriveToDiskPartition"} |
%{gwmi -Query "ASSOCIATORS OF {Win32_DiskPartition.DeviceID=`"$($_.DeviceID)`"} WHERE AssocClass = Win32_LogicalDiskToPartition"}
Write-Output $USBInfo | select #{n="Drive Letter";e={$_.DeviceID}},#{n="USB Name";e={$_.VolumeName}}
}
Function SelectDrive{
$usbs = GetUSBs
$usbs | Format-Table -AutoSize | Out-File $locBackupFolder\RemovableStorage.txt
$selectedDrive = $usbs | Out-GridView -OutputMode Single -Title "Choose USB Storage Device"
$selectedDrive | Out-File $locBackupFolder\Drive.txt -Encoding default
if ($SelectedDrive -eq $null) {
WarningCancel
RemoveBackupDrive
exit
} else {
write-output $selectedDrive.'Drive Letter'.Substring(0,2)
}
}
$backupDrive = SelectDrive
This version still writes to your two files, but it doesn't read either of them, instead it keeps the objects internally. This means you get a better display in the grid view as the grid's own column headings are used, and you don't get any selectable lines you don't want.
There's a lot going on in your script, I debugged through it myself and found that these lines are causing the problems
ShowUSBs | Out-File $locBackupFolder\RemovableStorage.txt
Get-Content $locBackupFolder\RemovableStorage.txt | Out-GridView -OutputMode Single -Title "Choose USB Storage Device" | Out-File $locBackupFolder\Drive.txt -Encoding default
$SelectedDrive = Get-Content "$locBackupFolder\Drive.txt"
You need to remove this block of code:
Out-File $locBackupFolder\Drive.txt -Encoding default
It is creating a new blank Drive.txt file, after which, Get-Content is being called on that empty file.
Therefore, it is getting the content of a blank file, which will always result in the $SelectedDrive variable being set to null.
I've added content to my Drive.txt file after this block has been removed and the $SelectedDrive variable is set to the content of the file as expected
This construct won't work as you seem to expect:
Get-Content file | Out-GridView | Out-File otherfile
Out-GridView doesn't pass the data back into the pipeline for further processing. If you need data in both a grid view and a file you need to use the Tee-Object cmdlet before Out-GridView:
Get-Content file | Tee-Object otherfile | Out-GridView
In your case:
Get-Content "$locBackupFolder\RemovableStorage.txt" |
Tee-Object "$locBackupFolder\Drive.txt" |
Out-GridView -OutputMode Single -Title 'Choose USB Storage Device'
If you want the user to select a particular Item, Out-GridView won't work at all, because it's for output only (hence the verb Out). You'll need a form with a list box to allow the user to select an item:
[void][System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")
[void][System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")
$frm = New-Object Windows.Forms.Form
$btn = New-Object Windows.Forms.Button
$btn.Location = New-Object Drawing.Size(10,220)
$btn.Size = New-Object Drawing.Size(75,23)
$btn.Text = "OK"
$btn.Add_Click({$frm.Close()})
$frm.Controls.Add($btn)
$list = New-Object Windows.Forms.ListBox
$list.Location = New-Object Drawing.Size(10,10)
$list.Size = New-Object Drawing.Size(260,40)
$list.Height = 200
Get-Content "$locBackupFolder\RemovableStorage.txt" | % {
$list.Items.Add($_)
} | Out-Null
$frm.Controls.Add($list)
$frm.Add_Shown({$frm.Activate()})
[void]$frm.ShowDialog()
$SelectedDrive = $list.SelectedItem