I am currently working on a project which is a music player but I am trying to make it work with SQL Server.
I have uploaded my music to SQL Server using queries , however the problem is retrieving them. The file type is varbinary(MAX). I tried to go an alternate route where i retrieve the binary in a text file and thereafter convert the file extension to .wav and play it through a media player while the local file being temporary.
Private Sub Online_Load(sender As Object, e As EventArgs) Handles MyBase.Load
Try
Using cn As SqlConnection = New SqlConnection("Server= DESKTOP-8KB6RLJ; Database= TESTdb; Integrated security = true ")
cn.Open()
Using cmd As SqlCommand = New SqlCommand()
cmd.Connection = cn
Dim qry As String
qry = String.Format("SELECT FileStreamCol From MUSIC")
cmd.CommandText = qry
cmd.CommandTimeout = 0
Dim oFileStream As System.IO.FileStream
oFileStream = New System.IO.FileStream("C:\3N3RGY\bytes.mp3", System.IO.FileMode.OpenOrCreate)
Using myReader As SqlDataReader = cmd.ExecuteReader(CommandBehavior.CloseConnection)
While (myReader.Read())
Dim data As Byte() = myReader(0)
oFileStream.Write(data, 0, data.Length)
End While
oFileStream.Close()
Dim myFiles As String()
myFiles = IO.Directory.GetFiles("C:\3N3RGY", "*.txt")
Dim newFilePath As String
For Each filepath As String In myFiles
newFilePath = filepath.Replace(".txt", ".mp3")
System.IO.File.Move(filepath, newFilePath)
AxWindowsMediaPlayer1.URL = ListBox1.SelectedItem
Next
End Using
End Using
End Using
Catch ex As Exception
End Try
End Sub
Related
I'm building a system that allows adding attachments. I have almost everything working, but any of the "open" formats from newer office files are coming out with the following error:
"We found a problem with some of the content in file.xlsx. Do you want us to recover as much as we can? IF you trust the source of this workbook, click yes."
clicking yes, usually will open the file, but I'd like to get past this entirely.
Files are stored in SQL Server, and uploaded and retrieved by stored procedure.
uploadFunction:
Protected Sub btnUploadAttachment_Click(sender As Object, e As EventArgs) Handles btnUploadAttachment.Click
Try
Dim fileCount As Integer = 0
For Each file As UploadedFile In filFileAttachment.UploadedFiles
fileCount += 1
Using conn As New SqlConnection With {.ConnectionString = SadSql},
cmd As New SqlCommand With {.CommandText = "RAD_ProjectTracker.dbo.spProject_Attachment_Add", .Connection = conn, .CommandType = CommandType.StoredProcedure, .CommandTimeout = SQLCommandTimeout}
Dim buffer As Byte()
Using s As Stream = file.InputStream
buffer = New Byte(s.Length) {}
s.Read(buffer, 0, buffer.Length)
End Using
cmd.Parameters.Add("#PROJECT_ID", SqlDbType.Int).Value = hfProjectID.Value
cmd.Parameters.Add("#VERSION_ID", SqlDbType.Int).Value = hfVersionID.Value
If Not hfJobID.Value.IsNullOrEmpty() Then cmd.Parameters.Add("#JOB_ID", SqlDbType.Int).Value = hfJobID.Value
cmd.Parameters.Add("#USER_ID", SqlDbType.Int).Value = Session("uid")
cmd.Parameters.Add("#ATTACHMENT", SqlDbType.VarBinary).Value = buffer
cmd.Parameters.Add("#ATTACHMENT_TYPE", SqlDbType.VarChar).Value = "ATTACH_TYPE_DOCUMENT"
cmd.Parameters.Add("#MIME_TYPE", SqlDbType.VarChar).Value = file.ContentType
cmd.Parameters.Add("#FILE_NAME", SqlDbType.VarChar).Value = file.FileName
If Not txtDescription.Text.IsNullOrEmpty() Then cmd.Parameters.Add("#DESCRIPTION", SqlDbType.VarChar).Value = txtDescription.Text
conn.Open()
cmd.ExecuteNonQuery()
conn.Close()
End Using
Next
ReturnToAttachments()
Catch ex As Exception
Dim s As String = ex.Message
End Try
End Sub
and the retrieval code:
Dim tsql As String = "Exec RAD_ProjectTracker.dbo.spProject_Attachments_Get #ATTACHMENT_ID=" & dataItem("ATTACHMENT_ID").Text
Dim dt As DataTable = GetSQLDataSet(tsql, True).TablesSafe(0)
If dt.Rows.Count > 0 Then
Dim dr As DataRow = dt.RowsSafe(0)
Dim file As Byte() = CType(dr("ATTACHMENT"), Byte())
Dim fileName As String = If(IsDBNull(dr("FILE_NAME")), "attachment", CStr(dr("FILE_NAME")))
Response.Clear()
Response.ClearHeaders()
Response.AddHeader("Content-Disposition", "attachment; filename=" + fileName)
Response.ClearContent()
Response.ContentType = dr("MIME_TYPE")
Response.BinaryWrite(file)
Response.End()
End If
I've tested .xls, .dox, .ppt, .png, .jpg, .txt, and a few others and they all work fine.
It's only the .xlsx, .docx, and .pptx that seem to be having issues.
I've googled around and haven't been able to find much. Is there an extra step to these documents? or perhaps i am doing something in the wrong order?
Any help would be appreciated.
I'm opening up a couple of CSV files and reading them in as a DataTable per the example I found here. The issue I am running into it the basic query I'm using to import the data is converting the column of IP addresses into Doubles. So I want to read in 10.0.0.1 and it shows up as 10.001. How can I get this column to read in as a string? I would like to not double process the file if I can.
Query I'm using is basic and is as follows:
SELECT * FROM [ComputerList.csv]
Here is my function to open and read the CSV file into a DataTable
Public Function OpenFile(ByVal strFolderPath as String, ByVal strQuery as String) as DataTable
Dim strConn as String = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & strFolderPath & ";Extended Propteries=""text; HDR=Yes;FMT=Delimited"""
Dim conn as OleDb.OleDbConnection = New OleDb.OleDbConnection(strConn)
Try
conn.Open()
Dim cmd as OleDb.OleDbCommand = New OleDb.OleDbCommand(strQuery, conn)
Dim da as OleDb.OleDbDataAdapter = New OleDb.OleDbDataAdapter()
da.SelectCommand = cmd
Dim ds as DataSet = New DataSet()
da.Fill(ds)
da.Dispose()
return ds.Tables(0)
Catch
return Nothing
Finally
conn.Close()
End Try
End Function
i use this to read csv's and force all to string.
Public Function convert_csv_to_data_table(ByVal File As String, ByVal separator As String) As DataTable
Dim dt As New DataTable
Dim firstLine As Boolean = True
If IO.File.Exists(File) Then
Using sr As New StreamReader(File)
While Not sr.EndOfStream
If firstLine Then
firstLine = False
Dim cols = sr.ReadLine.Split(separator)
For Each col In cols
dt.Columns.Add(New DataColumn(col, GetType(String)))
Next
Else
Dim data() As String = sr.ReadLine.Split(separator)
dt.Rows.Add(data.ToArray)
End If
End While
End Using
End If
Return dt
End Function
EDIT:- this will only work with a separator btw
ok I tried variations of everyone's suggested and have settled on a hybrid between all of them. My goal was to read in a CSV file manipulate it in a DataTable form and them write it back out. Some of my CSV files had multiple lines with in a cell and some had deliminators within a cell. Below you can find my hybrid solution that utilized TextFieldParser to read in the file and break it up.
Public Function OpenFile(ByVal File as String, NyVal delim as String) as DataTable
Dim dt as New DataTable()
Dim firstline as Boolean = True
Using MyReader as New Microsoft.VisualBasic.FileIO.TextFieldParser(File)
MyReader.TextFieldType = FileIO.FieldType.Delimited
MyReader.SetDelimiters(delim)
Dim currentRow as String()
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
If firstline
firstline = false
For Each col in currentRow
dt.Columns.Add(New DataColumn(col.ToString(), System.Type.GetType("System.String")))
Next
Else
dt.Rows.Add(currentRow.ToArray())
End If
Catch ex as Microsoft.VisualBasic.FileIO.MalformedLineException
Console.WriteLIne("Line " + ex.Message + " is not valid and will be skipped")
End Try
End While
End Using
return dt
End Function
My application is a web scraper (for the most part) that stores information in a database. I have 2 classes so far:
clsSpyder - This essentially rolls-up the scraper processes
clsDB - This does any database processes
My test program looks over all the URLs, scrapes, pushes into the database. It is pretty simple sequentially, but I would like to have say N number of threads running those processes (scrape and store). My sequential code is this:
Private Sub Button4_Click(sender As Object, e As EventArgs) Handles Button4.Click
'Grab List
Dim tDS As New DataSet
Dim tDB As New clsTermsDB
Dim tSpyder As New clsAGDSpyder
Dim sResult As New TermsRuns
'Grab a list of all URLS
tDS = tDB.GetTermsList(1)
Try
For Each Row As DataRow In tDS.Tables(0).Rows
rtbList.AppendText(Row("url_toBeCollected") & vbCrLf)
sResult = tSpyder.SpiderPage(Row("url_toBeCollected"))
'If nothing is found, do not store
If sResult.html <> "" And sResult.text <> "" Then
tDB.InsertScrape(Now(), sResult.html, sResult.text, Row("url_uid"), 1)
End If
Next
Exit Sub
Catch ex As Exception
MessageBox.Show(ex.Message)
End Try
End Sub
With that in mind and noting that I am passing variables to the SpiderPage and InsertScrape methods.. How could I implement threading? It's gotta be simple, but I feel like I have been googling and trying things for days without success :(
*** ADDED: SpiderPage method:
Public Function SpiderPage(PageURL As String) As TermsRuns
Dim webget As New HtmlWeb
Dim node As HtmlNode
Dim doc As New HtmlDocument
Dim docNOHTML As HtmlDocument
Dim uri As New Uri(PageURL)
Dim wc As HttpWebRequest = DirectCast(WebRequest.Create(uri.AbsoluteUri), HttpWebRequest)
Dim wcStream As Stream
wc.AllowAutoRedirect = True
wc.MaximumAutomaticRedirections = 3
'Set Headers
wc.UserAgent = "Mozilla/5.0 (Macintosh; I; Intel Mac OS X 11_7_9; de-LI; rv:1.9b4) Gecko/2012010317 Firefox/10.0a4"
wc.Headers.Add("REMOTE_ADDR", "66.83.101.5")
wc.Headers.Add("HTTP_REFERER", "66.83.101.5")
'Set HTMLAgility Kit Useragent Spoofing (not needed, I don't think)
webget.UserAgent = "Mozilla/5.0 (Macintosh; I; Intel Mac OS X 11_7_9; de-LI; rv:1.9b4) Gecko/2012010317 Firefox/10.0a4"
'Certification STuff
wc.UseDefaultCredentials = True
wc.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials
ServicePointManager.ServerCertificateValidationCallback = AddressOf AcceptAllCertifications
'Create Cookie Jar
Dim CookieJar As New CookieContainer
wc.CookieContainer = CookieJar
'Keep Alive Settings
wc.KeepAlive = True
wc.Timeout = &H7530
'Read the web page
Dim wr As HttpWebResponse = Nothing
Try
wcStream = wc.GetResponse.GetResponseStream
doc.Load(wcStream)
'Remove HTML from the document
docNOHTML = RemoveUnWantedTags(doc)
'Grab only the content inside the <body> tag
node = docNOHTML.DocumentNode.SelectSingleNode("//body")
'Output
SpiderPage = New TermsRuns
SpiderPage.html = node.InnerHtml
SpiderPage.text = node.InnerText
Return SpiderPage
Catch ex As Exception
'Something goes here when scraping returns an error
SpiderPage = New TermsRuns
SpiderPage.html = ""
SpiderPage.text = ""
End Try
End Function
*** Added InsertScrape:
Public Function InsertScrape(scrape_ts As DateTime, scrape_html As String, scrape_text As String, url_id As Integer, tas_id As Integer) As Boolean
Dim myCommand As MySqlClient.MySqlCommand
Dim dt As New DataTable
'Create ds/dt for fill
Dim ds As New DataSet
Dim dtbl As New DataTable
Try
'Set Connection String
myConn.ConnectionString = myConnectionString
'Push Command to Client Object
myCommand = New MySqlClient.MySqlCommand
myCommand.Connection = myConn
myCommand.CommandText = "spInsertScrape"
myCommand.CommandType = CommandType.StoredProcedure
myCommand.Parameters.AddWithValue("#scrape_ts", scrape_ts)
myCommand.Parameters("#scrape_ts").Direction = ParameterDirection.Input
myCommand.Parameters.AddWithValue("#scrape_html", scrape_html)
myCommand.Parameters("#scrape_html").Direction = ParameterDirection.Input
myCommand.Parameters.AddWithValue("#scrape_text", scrape_text)
myCommand.Parameters("#scrape_text").Direction = ParameterDirection.Input
myCommand.Parameters.AddWithValue("#url_id", url_id)
myCommand.Parameters("#url_id").Direction = ParameterDirection.Input
myCommand.Parameters.AddWithValue("#tas_id", tas_id)
myCommand.Parameters("#tas_id").Direction = ParameterDirection.Input
'Open Connection
myConn.Open()
myCommand.ExecuteNonQuery()
'Close Connection
myConn.Close()
InsertScrape = True
Catch ex As Exception
'Put Message Here
InsertScrape = False
MessageBox.Show(ex.Message)
End Try
End Function
thanks in advance.
Learning and testing using Sql FILESTREAM for a web app. A client uploads a large file form the web page which takes 'X' time and when fully uploaded shows 100% complete. However, very large files also take time for SqlFileStream to write to the file system so I want to spin off a thread to complete that part. The code I've got seems to work fine but no data ends up in the filestream file.
I'm wrapping the initial record creation in it's own transaction scope and using a separate transaction scope in the thread. In the threaded routine I have the appropriate PathName() and TransactionContext but I assume I'm missing something while using a thread.
I've commented out the normal SqlFileStream call which works fine. Can you see anything wrong with what I'm doing here?
Public Function StoreFileStream()
Dim json As New Dictionary(Of String, Object)
Dim parms As New FileStreamThreadParameters
If HttpContext.Current.Request.Files.Count > 0 Then
Dim file As HttpPostedFile = HttpContext.Current.Request.Files(0)
If "contentType" <> String.Empty Then
Dim fs As Stream = file.InputStream
Dim br As New BinaryReader(fs)
Dim noBytes As New Byte()
Try
Dim filePath As String = ""
Dim trxContext As Byte() = {}
Dim baseFileId As Integer
Using trxScope As New TransactionScope
Using dbConn As New SqlConnection(DigsConnStr)
Using dbCmd As New SqlCommand("ADD_FileStreamFile", dbConn)
dbConn.Open()
Using dbRdr As SqlDataReader = dbCmd.ExecuteReader(CommandBehavior.SingleRow)
dbRdr.Read()
If dbRdr.HasRows Then
filePath = dbRdr("Path")
trxContext = dbRdr("TrxContext")
baseFileId = dbRdr("BaseFileID")
End If
dbRdr.Close()
End Using
' Code below writes to file, but trying to offload this to a separate thread so user is not waiting
'Using dest As New SqlFileStream(filePath, trxContext, FileAccess.Write)
' fs.CopyTo(dest, 4096)
' dest.Close()
'End Using
End Using
dbConn.Close()
End Using
trxScope.Complete()
End Using ' transaction commits here, not in line above
parms.baseFileId = baseFileId
parms.fs = New MemoryStream
fs.CopyTo(parms.fs)
Dim fileUpdateThread As New Threading.Thread(Sub()
UpdateFileStreamThreaded(parms)
End Sub)
fileUpdateThread.Start()
json.Add("status", "success")
Catch ex As Exception
Elmah.ErrorSignal.FromCurrentContext().Raise(ex)
json.Add("status", "failure")
json.Add("msg", ex.Message)
json.Add("procedure", System.Reflection.MethodBase.GetCurrentMethod.Name)
End Try
Else
json.Add("status", "failure")
json.Add("msg", "Invalid file type")
json.Add("procedure", System.Reflection.MethodBase.GetCurrentMethod.Name)
End If
End If
Return json
End Function
Public Class FileStreamThreadParameters
Public Property baseFileId As Integer
Public fs As Stream
End Class
Private Sub UpdateFileStreamThreaded(parms As FileStreamThreadParameters)
Dim filePath As String = ""
Dim trxContext As Byte() = {}
Try
Using trxScope As New TransactionScope
Using dbConn As New SqlConnection(DigsConnStr)
Using dbCmd As New SqlCommand("SELECT FileBytes.PathName() 'Path', GET_FILESTREAM_TRANSACTION_CONTEXT() 'TrxContext' FROM FileStreamFile WHERE Id = " & parms.baseFileId, dbConn)
dbConn.Open()
Using dbRdr As SqlDataReader = dbCmd.ExecuteReader(CommandBehavior.SingleRow)
dbRdr.Read()
If dbRdr.HasRows Then
filePath = dbRdr("Path")
trxContext = dbRdr("TrxContext")
End If
dbRdr.Close()
Using dest As New SqlFileStream(filePath, trxContext, FileAccess.Write)
parms.fs.CopyTo(dest, 4096)
dest.Close()
End Using
End Using
End Using
dbConn.Close()
End Using
trxScope.Complete()
End Using
Catch ex As Exception
'Elmah.ErrorSignal.FromCurrentContext().Raise(ex)
End Try
End Sub
Obviously this is a complex issue. I actually got it to work with the code below. However I eventually abandoned using SQL FILESTREAM altogether due to too much complexity in getting it all set up.
This is an existing web application with the sql server on a different box. After I got the filestreaming to work the next hurdle was authentication setup. Filestream requires Integrated Security on your connection string. Even with windows authentication on our Intranet app, I could not get the web app to use the windows credentials when connecting to the database. It always seemed to use the computer name or the app pool service. I tried many many examples I found on the net and here to no avail. Even if I got that to work then I would want to use and Active Directory group over individual logins which looked to be another hurdle.
This app stores documents in a varbinary column so that full text search can be enabled at some point. The issue was with large files which are generally pictures or videos. Since you can't search text on those anyway the strategy now is to store those on the file system and all other searchable docs (.docx, .pptx, etc) will still be stored in the varbinary. I'm actually sad that I could not get filestream to work as it seems like the ideal solution. I'll come back to it some day but it really should not be so frickin complicated. :-(
The code I got working is:
Dim filePath As String = ""
Dim trxContext As Byte() = {}
Dim baseFileId As Integer
Using trxScope As New TransactionScope
Using dbConn As New SqlConnection(DigsFSConnStr)
Using dbCmd As New SqlCommand("NEW_FileStreamBaseFile", dbConn)
dbCmd.CommandType = CommandType.StoredProcedure
dbCmd.Parameters.AddWithValue("#Title", fileDesc)
dbCmd.Parameters.AddWithValue("#Summary", summary)
dbCmd.Parameters.AddWithValue("#Comments", comments)
dbCmd.Parameters.AddWithValue("#FileName", uploadedFileName)
dbCmd.Parameters.AddWithValue("#ContentType", contentType)
dbCmd.Parameters.AddWithValue("#FileExt", ext)
'dbCmd.Parameters.AddWithValue("#FileBytes", noBytes) ' now that were offloading the file byte storage to a thread
dbCmd.Parameters.AddWithValue("#UploadedByResourceID", uploadedByResourceID)
dbCmd.Parameters.AddWithValue("#UploadedByShortName", uploadedByShortName)
dbCmd.Parameters.AddWithValue("#FileAuthor", fileAuthor)
dbCmd.Parameters.AddWithValue("#TagRecID", tagRecID)
dbCmd.Parameters.AddWithValue("#UserID", samAccountName)
dbCmd.Parameters.AddWithValue("#FileDate", fileDate)
dbCmd.Parameters.AddWithValue("#FileType", fileType)
dbCmd.Parameters.AddWithValue("#FileTypeRecID", fileTypeRecId)
' Save to file system too for xod conversion
file.SaveAs(HttpContext.Current.Server.MapPath("~/files/uploaded/") & uploadedFileName)
dbConn.Open()
Using dbRdr As SqlDataReader = dbCmd.ExecuteReader(CommandBehavior.SingleRow)
dbRdr.Read()
If dbRdr.HasRows Then
filePath = dbRdr("Path")
trxContext = dbRdr("TrxContext")
json.Add("baseFileId", dbRdr("BaseFileID"))
virtualFileRecId = dbRdr("VirtualFileRecID")
dbStatus = dbRdr("status")
If dbStatus = "failure" Then
json.Add("msg", dbRdr("msg"))
End If
End If
dbRdr.Close()
End Using
' Prepare and start Task thread to write the file
If dbStatus = "success" Then
bytes = br.ReadBytes(fs.Length)
Dim task As New System.Threading.Tasks.Task(
Sub()
UpdateNewFileStreamBytes(virtualFileRecId, bytes)
End Sub)
task.Start()
json.Add("status", "success")
Else
json.Add("status", "failure")
End If
End Using
dbConn.Close()
End Using
trxScope.Complete()
End Using ' transaction commits here, not in line above
With the task procedure of:
Private Sub UpdateNewFileStreamBytes(virtualFileRecId As Integer, fileBytes As Byte())
Dim filePath As String = ""
Dim trxContext As Byte() = {}
Try
Using trxScope As New TransactionScope
Using dbConn As New SqlConnection(DigsFSConnStr)
Using dbCmd As New SqlCommand("UPD_FileStreamBaseFile", dbConn)
dbCmd.CommandType = CommandType.StoredProcedure
dbCmd.Parameters.AddWithValue("#VirtualFileRecID", virtualFileRecId)
dbConn.Open()
Using dbRdr As SqlDataReader = dbCmd.ExecuteReader(CommandBehavior.SingleRow)
dbRdr.Read()
If dbRdr.HasRows Then
filePath = dbRdr("Path")
trxContext = dbRdr("TrxContext")
End If
dbRdr.Close()
Using dest As New SqlFileStream(filePath, trxContext, FileAccess.Write)
dest.Write(fileBytes, 0, fileBytes.Length)
dest.Close()
End Using
End Using
End Using
dbConn.Close()
End Using
trxScope.Complete()
End Using
Catch ex As Exception
Elmah.ErrorSignal.FromCurrentContext().Raise(ex)
End Try
I have a code which process each file in a folder and send the data in the file to SQL Database and then deletes the file. But often i get this Exception "The process cannot access the file 'file.csv' because it is being used by another process." Please can anyone point me out in the right direction, will really appreciate that.
The code is shown below :
Dim dirinfo As DirectoryInfo
Dim allFiles() As FileInfo
dirinfo = New DirectoryInfo("E:\SQLUPDATE\CAC")
allFiles = dirinfo.GetFiles("*.csv")
If allFiles.Length <> 0 Then
Try
For Each fl As FileInfo In allFiles
'MsgBox(fl.FullName.ToString())
Dim con As SqlConnection = New SqlConnection(SQL_con2)
Dim sr As StreamReader = New StreamReader(fl.FullName)
Dim line As String = sr.ReadLine
Dim value() As String = line.Split(Microsoft.VisualBasic.ChrW(44))
Dim dt As DataTable = New DataTable
Dim row As DataRow
For Each dc As String In value
dt.Columns.Add(New DataColumn(dc))
Next
While Not sr.EndOfStream
value = sr.ReadLine.Split(Microsoft.VisualBasic.ChrW(44))
If (value.Length = dt.Columns.Count) Then
row = dt.NewRow
row.ItemArray = value
dt.Rows.Add(row)
End If
End While
Dim bc As SqlBulkCopy = New SqlBulkCopy(con.ConnectionString, SqlBulkCopyOptions.TableLock)
bc.DestinationTableName = "[DB].[dbo].[CAC_LData]"
bc.BatchSize = dt.Rows.Count
con.Open()
bc.WriteToServer(dt)
bc.Close()
con.Close()
sr.Close()
System.IO.File.Delete(fl.FullName)
sr.Dispose()
Next
Catch ex As Exception
MsgBox(ex.Message)
End Try
End If