I am new to Amazon S3 and after searching through Amazon & many sites I have made some progress.
I am creating a windows From application to upload the files. I tried with following code and it does not upload the file and I don't get any errors.
If I comment the .withKey property than file is uploaded the root of the bucket, but instead I would like to upload to a specific folder within a folder.
Try
Dim fileTransferUtility As New TransferUtility(client)
Dim request As TransferUtilityUploadRequest = New TransferUtilityUploadRequest
Dim S3_KEY As String = "MitsTest/CVs" 'I want to upload file to CVs folder
With request
.BucketName = "mybucketname"
.WithKey(S3_KEY)
.WithFilePath("C:\Users\mitesh\Desktop\s3.txt")
.WithTimeout(FIVE_MINUTES)
End With
fileTransferUtility.Upload(request)
Catch ex As Exception
Finally
End Try
I have also tried using following and same result, I dont get any error but file does not get uploaded.
Dim Filepath = "C:\Users\mitesh\Desktop\s3.txt"
Dim FolderName = "MitsTest/CVs/"
Dim PutRequest As New PutObjectRequest()
Try
With PutRequest
.WithBucketName("mybucketname")
'.WithKey(FolderName + "/" + FilePath)
.WithKey(FolderName)
.WithFilePath(FilePath)
.StorageClass = S3StorageClass.Standard
.ServerSideEncryptionMethod = ServerSideEncryptionMethod.None
'.WithContentBody(String.Empty)
Dim response As S3Response = client.PutObject(PutRequest)
response.Dispose()
End With
Catch ex As Exception
Stop
End Try
if I uncomment this line '.WithKey(FolderName + "/" + FilePath) and remove the forward slash(/) from folder name, then I get "The request signature we calculated does not match the signature you provided. Check your key and signing method." error.
What am i missing here or doing wrong? Any help would be really appreciated.
regards
Mitesh
The reason you get the signing error is that the back slash characters in "C:\users.." are invalid for key names. You should either replace them with forward slash or use a different value for the key name.
In your first example, you are actually trying to upload to the "MitsTest/CVs" key. You would need to append the name of the file like this:
.WithKey(S3_KEY+ "/" + "s3.txt")
Related
I need to delete a Sharepoint file using the Script Task from SSIS. In Visual Basic, I've tried using the SPListItemCollection with imports Microsoft.Sharepoint but it doesn't recognize the namespace. I didn't find lots of threads on this subject or what I've found wasn't related with script task, so any help will be really appreciated. Many thanks
Update based on #Hadi answer
Thanks Hadi for your answer. I've given up the idea of using SPListCollection as it seems too complicated. Instead I'm trying to delete the file after it is downloaded from Sharepoint to the local folder. I would need help at the line that actually deletes the file. Here is the code:
Public Sub Main()
Try
' get location of local folder
Dim dir As DirectoryInfo = New DirectoryInfo(Dts.Variables("DestP").Value.ToString())
If dir.Exists Then
' Create the filename for local storage
Dim file As FileInfo = New FileInfo(dir.FullName & "\" & Dts.Variables("FileName").Value.ToString())
If Not file.Exists Then
' get the path of the file to download
Dim fileUrl As String = Dts.Variables("SHP_URL").Value.ToString()
If fileUrl.Length <> 0 Then
Dim client As New WebClient()
If Left(fileUrl, 4).ToLower() = "http" Then
'download the file from SharePoint
client.Credentials = New System.Net.NetworkCredential(Dts.Variables("$Project::UserN").Value.ToString(), Dts.Variables("$Project::Passw").Value.ToString())
client.DownloadFile(fileUrl.ToString() & "/" & Dts.Variables("FileName").Value.ToString(), file.FullName)
Else
System.IO.File.Copy(fileUrl.ToString() & Dts.Variables("FileName").Value.ToString(), file.FullName)
End If
'delete file from Sharepoint
client.(fileUrl.ToString() & "/" & Dts.Variables("FileName").Value.ToString(), file.FullName).delete()
Else
Throw New ApplicationException("EncodedAbsUrl variable does not contain a value!")
End If
End If
Else
Throw New ApplicationException("No ImportFolder!")
End If
Catch ex As Exception
Dts.Events.FireError(0, String.Empty, ex.Message, String.Empty, 0)
Dts.TaskResult = ScriptResults.Failure
End Try
Dts.TaskResult = ScriptResults.Success
End Sub
Update 1 - Delete using FtpWebRequest
You cannot delete file using WebClient class. You can do that using FtpWebRequest class. And send a WebRequestMethods.Ftp.DeleteFile request as mentioned in the link below:
How To Delete a File From FTP Server in C#
It should work with Sharepoint also.
Here is the function in VB.NET
Private Function DeleteFile(ByVal fileName As String) As String
Dim request As FtpWebRequest = CType(WebRequest.Create(fileUrl.ToString() & "/" & fileName), FtpWebRequest)
request.Method = WebRequestMethods.Ftp.DeleteFile
request.Credentials = New NetworkCredential(Dts.Variables("$Project::UserN").Value.ToString(), Dts.Variables("$Project::Passw").Value.ToString())
Using response As FtpWebResponse = CType(request.GetResponse(), FtpWebResponse)
Return response.StatusDescription
End Using
End Function
You should replace the following line:
client.(fileUrl.ToString() & "/" & Dts.Variables("FileName").Value.ToString(), file.FullName).delete()
With
DeleteFile(Dts.Variables("FileName").Value.ToString())
Also you may use the following credentials:
request.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
References
How to pass credentials to httpwebrequest for accessing SharePoint Library
How To Delete a File From FTP Server in C#
Downloading all files in a FTP folder and then deleting them
Deleting file from FTP in C#
How To Delete a File From FTP Server in C#
Initial Answer
I was searching for a similar issue from a while, it looks like you cannot delete a Sharepoint file in SSIS using a File System Task or Execute Process Task, the only way is using a Script Task. There are many links online describing this process such as:
how to delete or remove only text files from share point in C# or SSIS script?
Fastest way to delete all items with C#
Deleting files programatically
Deleting all the items from a large list in SharePoint
Concerning the problem that you have mentioned, i think you should make sure that Microsoft.Sharepoint.dll is added as a reference inside the Script Task. If so try using Microsoft.Sharepoint.SPListItemCollection instead of SPListItemCollection.
Thanks #Hadi for your help.
For me it didn't work with FTPWebResponse.
It worked with HttpWebRequest. Here is the script:
Dim request As System.Net.HttpWebRequest = CType(WebRequest.Create(fileUrl.ToString() & "/" & Dts.Variables("FileName").Value.ToString()), HttpWebRequest)
request.Credentials = New System.Net.NetworkCredential(Dts.Variables("$Project::UserN").Value.ToString(), Dts.Variables("$Project::Passw").Value.ToString())
request.Method = "DELETE"
Dim response As System.Net.HttpWebResponse = CType(request.GetResponse(), HttpWebResponse)
i'm working in a console app that is going to be executed in a server every week, basically it generates a report in excel and then it has to upload it to a folder in dropbox, i've trying a lot of stuff for that last part i finally got this code that does not work but doesn't throw any exception (before i had one that throw and invalid folder format)
Dim _path As String
_path = "/Pruebas/" & Path.GetFileName(FilePath)
Try
Dim rawData = File.ReadAllBytes(FilePath)
Dim str = System.Text.Encoding.Default.GetString(rawData)
Using mem = New MemoryStream(Encoding.UTF8.GetBytes(str))
Dim Up = I.Files.UploadAsync(_path, body:=mem)
MsgBox("Successfully Uploaded")
End Using
Catch ex As Exception
MsgBox(ex.Message)
End Try
it doesn't throw exception but also doens't work, any help i'll be thankfull.
try this.
Dim Up = I.Files.UploadAsync(_path, WriteMode.Overwrite.Instance, body:=mem)
Link to documentation: http://dropbox.github.io/dropbox-sdk-dotnet/html/M_Dropbox_Api_Files_Routes_FilesUserRoutes_UploadAsync_1.htm
I'm trying to upload a file to dropbox. The file gets uploaded and code works when i'm running locally. But file never gets uploaded when published on the server. I'm having the following error .
Could not load type 'System.Security.Authentication.SslProtocols' from assembly 'System.Net.Primitives, Version=4.0.0.0, Culture=neutral, PublicKeyToken=*********'.
I'm trying the below code sample to upload it
public sub uploadtodropbox()
Using httpclient As New HttpClient
httpclient.Timeout = TimeSpan.FromMinutes(20)
Dim _DropboxClient = New DropboxClient(ApiKeyDropBox, config)
Dim resultstring As String = UploadToDB(_DropboxClient, dropboxdir, docName, fileBytes)
_DropboxClient.Dispose()
If Not resultstring = "RanToCompletion" Then
ErrorMsg &= "The following error occured: " & resultstring
End If
End Using
End Sub
and this is the funtion that is uploading to Dropbox
Private Function UploadToDB(_DropboxClient As DropboxClient, Directory As String, Filename As String, Content As [Byte]()) As String
Try
Dim result As String = "Unknown"
Dim trycnt As String = 0
Dim tryLimit As String = 20
Dim respnse As Task(Of FileMetadata)
Using _mStream = New MemoryStream(Content)
respnse = _DropboxClient.Files.UploadAsync(Convert.ToString(Directory & Convert.ToString("/")) & Filename, WriteMode.Overwrite.Instance, body:=_mStream)
While Not respnse.IsCompleted And trycnt < tryLimit
Threading.Thread.Sleep(1000)
trycnt += 1
result = respnse.Status.ToString()
End While
UploadToDB = result
End Using
End try
END function.
I have tried without using the httpclient then i am getting this error :
The type initializer for 'Dropbox.Api.DropboxRequestHandlerOptions' threw an exception.
Thank you
I go the answer . I have system.Net.primitives assembly that is creating a problem uploading the file to Dropbox. All i need to do is delete the reference and also the System.Net.Http has versions have been set wrong in the web.config file. Once i set that in the configuration everything works fine.
For the certificate I have set the on post back to validate and allow the certificates generated in the event handler.
Quick question about downloading Blobs from azure storage. I'm a little lost on how this works exactly, here's my code:
Try
Dim accountname As String = "macroqc"
Dim accountkey As String = My.Settings.Storagekey1
Dim creds As StorageCredentials = New StorageCredentials(accountname, accountkey)
Dim account As CloudStorageAccount = New CloudStorageAccount(creds, useHttps:=True)
Dim client = account.CreateCloudBlobClient()
Dim container As CloudBlobContainer = client.GetContainerReference("smallequipment1certifications")
Dim blob As CloudBlockBlob = container.GetBlockBlobReference(filenamelbl.Text)
Using MemoryStream = New MemoryStream()
blob.DownloadToStream(MemoryStream)
My.Settings.downloadsource = System.Text.Encoding.UTF8.GetString(MemoryStream.ToArray())
equipmentpdf.src = filenamelbl.Text
End Using
Catch ex As Exception
MessageBox.Show("Sorry an error has occured while opening your file: " & Environment.NewLine & ex.ToString, "Download Error", MessageBoxButtons.OK, MessageBoxIcon.Error)
End Try
So I am getting an error when I run this which looks like:
So when it uploads the file to Azure storage it saves blob.Uri.AbsoluteUri & blob.Uri.AbsolutePath to the database. Am I going down the right path using this method to download the file? The Blobs are private do I need an SAS to download? I am confused and am having a really hard time finding useful documenation on this! Can someone help point me in the right direction please!!
Thanks everyone!
The reason you're getting this error is because GetBlockBlobReference expects the name of the blob (and not the full blob URL) and it creates a URL based on that. Since you're passing the complete URL, it still creates a URL but includes the blob URL. If you put a breakpoint and check the blob object's URL property, you will see something like:
https://accountname.blob.core.windows.net/containername/https://accountname.blob.core.windows.net/containername/filename
Since the blob by the name https://accountname.blob.core.windows.net/containername/filename does not exist in the container, you're getting a 404 (Not Found) error. Please provide just the file name in GetBlockBlobReference and things should work just fine.
I'd like to know the last-modifed date of a remote file (defined via url).
And only download it, if it's newer than my locally stored one.
I managed to do that for local files, but can't find a solution to do that for remote files (without downloading them)
working:
Dim infoReader As System.IO.FileInfo = My.Computer.FileSystem.GetFileInfo("C:/test.txt")
MsgBox("File was last modified on " & infoReader.LastWriteTime)
not working:
Dim infoReader As System.IO.FileInfo = My.Computer.FileSystem.GetFileInfo("http://google.com/robots.txt")
MsgBox("File was last modified on " & infoReader.LastWriteTime)
I'd love to have a solution which will only have to download the headers of a file
You can use the System.Net.Http.HttpClient class to fetch the last modified date from the server. Because it's sending a HEAD request, it will not fetch the file contents:
Dim client = New HttpClient()
Dim msg = New HttpRequestMessage(HttpMethod.Head, "http://google.com/robots.txt")
Dim resp = client.SendAsync(msg).Result
Dim lastMod = resp.Content.Headers.LastModified
You could also use the If-Modified-Since request header with a GET request. This way the response should be 304 - Not Modified if the file has not been changed (no file content sent), or 200 - OK if the file has been changed (and the contents of the file will be sent in the response), although the server is not required to honor this header.
Dim client = New HttpClient()
Dim msg = New HttpRequestMessage(HttpMethod.Get, "http://google.com/robots.txt")
msg.Headers.IfModifiedSince = DateTimeOffset.UtcNow.AddDays(-1) ' use the date of your copy of the file
Dim resp = client.SendAsync(msg).Result
Select Case resp.StatusCode
Case HttpStatusCode.NotModified
' Your copy is up-to-date
Case HttpStatusCode.OK
' Your copy is out of date, so save it
File.WriteAllBytes("C:\robots.txt", resp.Content.ReadAsByteArrayAsync.Result)
End Select
Note the use of .Result, since I was testing in a console application - you should probably await instead.
If the server offers it, you can get it through the HTTP header Last-Modified property. But your still stuck at downloading the full file.
You could get it through FTP.
See if the server allows you to see the list of files in a folder.
If the website offer the date somewhere that you could pull through screen scrapping.
I know this is a little bit old question but, there's still a better answer.
Dim req As WebRequest = HttpWebRequest.Create("someurl")
req.Method = "HEAD"
Dim resp As WebResponse = req.GetResponse()
Dim remoteFileLastModified As String = resp.Headers.Get("Last-Modified")
Dim remoteFileLastModifiedDateTime As DateTime
If DateTime.TryParse(remoteFileLastModified, remoteFileLastModifiedDateTime) Then
MsgBox("Date Last Modified:" + remoteFileLastModifiedDateTime.ToString("d MMMM yyyy dddd HH:mm:ss"))
Else
MsgBox("could not determine")
End If