Deserialize json string and insert in SQL DB - vb.net

I have a function which gets json string from jsonblob, deserializers it and returns in dataview. I want to know how can i insert the data from the dataview into my sql database using vb.net
My function is as follows
Public Function GetJsonString() As DataView
Dim dsContactMeta As DataSet = Nothing
Dim dv As DataView = Nothing
'url declaration for REST_ContactMeta
Dim REST_ContactMetaURL As Uri = New Uri("http://jsonblob.com/api/jsonBlob/5420df11e4b00ad1f05ed29b")
Dim strJSONData As String = ""
Dim response As HttpWebResponse
Dim reader As StreamReader
Try
Dim httpWebRequest As HttpWebRequest = WebRequest.Create(REST_ContactMetaURL)
httpWebRequest.Method = WebRequestMethods.Http.Get
httpWebRequest.ContentType = "application/json"
response = httpWebRequest.GetResponse()
reader = New StreamReader(response.GetResponseStream)
strJSONData = reader.ReadToEnd
strJSONData = "[" & strJSONData & "]" 'had to wrap the raw JSON data with brackets for parsing purposes
Dim myXmlNode As System.Xml.XmlNode = JsonConvert.DeserializeXmlNode("{""root"":" + strJSONData + "}", "root") 'error is thrown if the JSON does not have a root element
' Dim myXmlNode As System.Xml.XmlNode = JsonConvert.DeserializeXmlNode(strJSONData)
dsContactMeta = New DataSet("Test")
Try
dsContactMeta.ReadXml(New XmlNodeReader(myXmlNode))
If dsContactMeta.Tables("root") Is Nothing Or dsContactMeta.Tables("root").Rows.Count = 0 Then
Return Nothing 'in case of error return nothing
Else
dsContactMeta.Tables("root").Columns("job_id").SetOrdinal(0) 'setting the field name as the first column
dv = New DataView
dv.Table = dsContactMeta.Tables("root")
Return dv
End If
Catch ex As Exception
Return Nothing
End Try
Catch ex As Exception
Return Nothing 'if for any reason the API has failed connecting to the server or other unknown error then return nothing
End Try
End Function

Insert record into database from DataView, for that you need to read DataView:
For Each rowView As DataRowView In dataView
Debug.Write(item & " ")
Dim row as DataRow = rowView.Row;
// Do something //
//insert into tablename(col1,col2,..) values(row['col1'],row['col2'],..)
Next

Related

Read the Response and stored to database using Microsoft SMO (Server Management Objects)

Im trying to call to an URL
And than i will get response into XML format and based what Response i will get it will create Tables successfully into database using Microsoft SMO (Server Management Objects) which Allowing me to create and generate databases and tables. but for some reason it coludnt insert any data to tables and they are NULL Screenshot
XML:
<ICECAT-interface xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://data.icecat.biz/xsd/files.index.xsd">
<files.index Generated="20200219011050">
<file path="export/freexml.int/EN/1424.xml" Product_ID="1424" Updated="20200218012414" Quality="ICECAT" Supplier_id="1" Prod_ID="C4811A" Catid="846" On_Market="1" Model_Name="11" Product_View="513730" HighPic="http://images.icecat.biz/img/gallery/1424_6912543175.jpg" HighPicSize="2478427" HighPicWidth="2598" HighPicHeight="3276" Date_Added="20051023000000" Limited="No" >
<M_Prod_ID Supplier_id="11" Supplier_name="Cisco">C4811A</M_Prod_ID>
<M_Prod_ID>Tempcomp3109</M_Prod_ID>
<Country_Markets>
<Country_Market Value="NL"/>
<Country_Market Value="BE"/>
<Country_Market Value="FR"/>
<Country_Market Value="UA"/>
<Country_Market Value="GB"/>
<Country_Market Value="DE"/>
<Country_Market Value="BG"/>
</Country_Markets>
</file>
</files.index>
</ICECAT-interface>
Module:
Class Module1
Public Shared Sub Main()
Dim url As String = "http://data.Icecat.biz/export/freexml/EN/daily.index.xml"
ProcessXMLFeedURL(url)
End Sub
Public Shared Function ProcessXMLFeedURL(MyURL As String) As Boolean
Dim OK As Boolean = False
Try
Dim rssReq As WebRequest = WebRequest.Create(MyURL)
Dim username As String = ""
Dim password As String = ""
Dim encoded As String = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password))
rssReq.Headers.Add("Authorization", "Basic " + encoded)
'//Get the WebResponse
Dim rep As WebResponse = rssReq.GetResponse()
'//Read the Response in a XMLTextReader
Dim xtr As XmlTextReader = New XmlTextReader(rep.GetResponseStream())
'// Set up the connection to the SQL server
Dim MyConnectionString As String = "Data Source=......"
Dim Connection As SqlConnection = New SqlConnection(MyConnectionString)
Dim MyServer As Server = New Server(New ServerConnection(Connection))
Dim db As Database = New Database(MyServer, "xxxxx")
db.Create()
'//Create a new DataSet
Dim ds As DataSet = New DataSet()
'//Read the Response into the DataSet
ds.ReadXml(xtr)
'// Parse tables
For i As Integer = 0 To ds.Tables.Count - 1
Dim Mytable As Table
Dim MyTableName As String = ds.Tables(i).TableName
If Not HaveTable(MyConnectionString, MyTableName) Then
'// Create the table
Try
Mytable = New Table(db, MyTableName)
Catch ex As Exception
Dim ii As Integer = 0
End Try
'// Create the columns
Dim Mycolumn As Column = New Column()
For Each dc As DataColumn In ds.Tables(i).Columns
Mycolumn = New Column(Mytable, dc.ColumnName)
Mycolumn.DataType = getdatatype(dc.DataType.ToString)
Mytable.Columns.Add(Mycolumn)
Next
Mytable.Create()
Dim PrimaryKeys() As DataColumn = ds.Tables(i).PrimaryKey
Dim PrimaryKey As DataColumn
For Each PrimaryKey In PrimaryKeys
Dim Myindex As Index = New Index(Mytable, PrimaryKey.ColumnName)
Myindex.IndexKeyType = IndexKeyType.DriPrimaryKey
Myindex.IndexedColumns.Add(New IndexedColumn(Myindex, PrimaryKey.ColumnName))
Mytable.Indexes.Add(Myindex)
Next
End If
Using MyConnection As SqlConnection = New SqlConnection(MyConnectionString)
MyConnection.Open()
Using bulkcopy As SqlBulkCopy = New SqlBulkCopy(MyConnection)
bulkcopy.DestinationTableName = "[" & MyTableName & "]"
Try
bulkcopy.WriteToServer(ds.Tables(i))
Catch ex As Exception
Dim iw As Integer = 0
End Try
End Using
MyConnection.Close()
End Using
Next
Catch ex As Exception
Throw ex '// Do errorhanddling here
End Try
Return OK
End Function
Shared Function getdatatype(Mydatatype As String) As DataType
Dim dty As DataType = Nothing
Select Case Mydatatype
Case Is = "System.Decimal"
dty = DataType.Decimal(2, 18)
Case Is = "System.String"
dty = DataType.VarChar(500)
Case Is = "System.Int32"
dty = DataType.Int
End Select
Return dty
End Function
Shared Function HaveTable(MyConnectionString As String, TableName As String) As Boolean
Dim OK As Boolean = False
Try
Dim dbConn As New SqlConnection(MyConnectionString)
dbConn.Open()
Dim restrictions(3) As String
restrictions(2) = TableName
Dim dbTbl As DataTable = dbConn.GetSchema("Tables", restrictions)
If dbTbl.Rows.Count > 0 Then
OK = True
End If
dbTbl.Dispose()
dbConn.Close()
dbConn.Dispose()
Catch ex As Exception
Dim ss As Integer = 0
End Try
Return OK
End Function
End Class
Can anyone please help me!

vb.net proper way to thread this application

My application is a web scraper (for the most part) that stores information in a database. I have 2 classes so far:
clsSpyder - This essentially rolls-up the scraper processes
clsDB - This does any database processes
My test program looks over all the URLs, scrapes, pushes into the database. It is pretty simple sequentially, but I would like to have say N number of threads running those processes (scrape and store). My sequential code is this:
Private Sub Button4_Click(sender As Object, e As EventArgs) Handles Button4.Click
'Grab List
Dim tDS As New DataSet
Dim tDB As New clsTermsDB
Dim tSpyder As New clsAGDSpyder
Dim sResult As New TermsRuns
'Grab a list of all URLS
tDS = tDB.GetTermsList(1)
Try
For Each Row As DataRow In tDS.Tables(0).Rows
rtbList.AppendText(Row("url_toBeCollected") & vbCrLf)
sResult = tSpyder.SpiderPage(Row("url_toBeCollected"))
'If nothing is found, do not store
If sResult.html <> "" And sResult.text <> "" Then
tDB.InsertScrape(Now(), sResult.html, sResult.text, Row("url_uid"), 1)
End If
Next
Exit Sub
Catch ex As Exception
MessageBox.Show(ex.Message)
End Try
End Sub
With that in mind and noting that I am passing variables to the SpiderPage and InsertScrape methods.. How could I implement threading? It's gotta be simple, but I feel like I have been googling and trying things for days without success :(
*** ADDED: SpiderPage method:
Public Function SpiderPage(PageURL As String) As TermsRuns
Dim webget As New HtmlWeb
Dim node As HtmlNode
Dim doc As New HtmlDocument
Dim docNOHTML As HtmlDocument
Dim uri As New Uri(PageURL)
Dim wc As HttpWebRequest = DirectCast(WebRequest.Create(uri.AbsoluteUri), HttpWebRequest)
Dim wcStream As Stream
wc.AllowAutoRedirect = True
wc.MaximumAutomaticRedirections = 3
'Set Headers
wc.UserAgent = "Mozilla/5.0 (Macintosh; I; Intel Mac OS X 11_7_9; de-LI; rv:1.9b4) Gecko/2012010317 Firefox/10.0a4"
wc.Headers.Add("REMOTE_ADDR", "66.83.101.5")
wc.Headers.Add("HTTP_REFERER", "66.83.101.5")
'Set HTMLAgility Kit Useragent Spoofing (not needed, I don't think)
webget.UserAgent = "Mozilla/5.0 (Macintosh; I; Intel Mac OS X 11_7_9; de-LI; rv:1.9b4) Gecko/2012010317 Firefox/10.0a4"
'Certification STuff
wc.UseDefaultCredentials = True
wc.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials
ServicePointManager.ServerCertificateValidationCallback = AddressOf AcceptAllCertifications
'Create Cookie Jar
Dim CookieJar As New CookieContainer
wc.CookieContainer = CookieJar
'Keep Alive Settings
wc.KeepAlive = True
wc.Timeout = &H7530
'Read the web page
Dim wr As HttpWebResponse = Nothing
Try
wcStream = wc.GetResponse.GetResponseStream
doc.Load(wcStream)
'Remove HTML from the document
docNOHTML = RemoveUnWantedTags(doc)
'Grab only the content inside the <body> tag
node = docNOHTML.DocumentNode.SelectSingleNode("//body")
'Output
SpiderPage = New TermsRuns
SpiderPage.html = node.InnerHtml
SpiderPage.text = node.InnerText
Return SpiderPage
Catch ex As Exception
'Something goes here when scraping returns an error
SpiderPage = New TermsRuns
SpiderPage.html = ""
SpiderPage.text = ""
End Try
End Function
*** Added InsertScrape:
Public Function InsertScrape(scrape_ts As DateTime, scrape_html As String, scrape_text As String, url_id As Integer, tas_id As Integer) As Boolean
Dim myCommand As MySqlClient.MySqlCommand
Dim dt As New DataTable
'Create ds/dt for fill
Dim ds As New DataSet
Dim dtbl As New DataTable
Try
'Set Connection String
myConn.ConnectionString = myConnectionString
'Push Command to Client Object
myCommand = New MySqlClient.MySqlCommand
myCommand.Connection = myConn
myCommand.CommandText = "spInsertScrape"
myCommand.CommandType = CommandType.StoredProcedure
myCommand.Parameters.AddWithValue("#scrape_ts", scrape_ts)
myCommand.Parameters("#scrape_ts").Direction = ParameterDirection.Input
myCommand.Parameters.AddWithValue("#scrape_html", scrape_html)
myCommand.Parameters("#scrape_html").Direction = ParameterDirection.Input
myCommand.Parameters.AddWithValue("#scrape_text", scrape_text)
myCommand.Parameters("#scrape_text").Direction = ParameterDirection.Input
myCommand.Parameters.AddWithValue("#url_id", url_id)
myCommand.Parameters("#url_id").Direction = ParameterDirection.Input
myCommand.Parameters.AddWithValue("#tas_id", tas_id)
myCommand.Parameters("#tas_id").Direction = ParameterDirection.Input
'Open Connection
myConn.Open()
myCommand.ExecuteNonQuery()
'Close Connection
myConn.Close()
InsertScrape = True
Catch ex As Exception
'Put Message Here
InsertScrape = False
MessageBox.Show(ex.Message)
End Try
End Function
thanks in advance.

DTD validation issue

I'm using the following code to validate an XML file:
'DTD validation
Dim doc As XmlDocument = New XmlDocument()
Dim dtdDoc As String = ftpPath & "\" & dtdDocument
AppendText(vbCrLf)
Try
doc.Load(fileName)
Catch ex As Exception
AppendText(ex.Message)
ShowFailMessage()
End Try
Dim isXmlValid As Boolean = True
Dim xmlValMsg As New StringBuilder()
Dim sw As New StringWriter()
doc.Save(sw)
doc.Save(fileName)
Dim stream As Stream = New FileStream(fileName, FileMode.Open)
Dim settings As New XmlReaderSettings()
settings.ValidationType = ValidationType.DTD
settings.DtdProcessing = DtdProcessing.Parse
AddHandler settings.ValidationEventHandler, AddressOf ValidationCallBack
Dim r As XmlReader = XmlReader.Create(stream, settings, "R:\ftproot\graphics\" & dtdDocument)
While r.Read()
End While
r.Close()
stream.Close()
stream.Dispose()
Private Shared Sub ValidationCallBack(sender As Object, e As ValidationEventArgs)
AppendText("DTD Validation Error: " & e.Message)
isDTDValidated = False
End Sub
I am getting the following error from this validation code:
"The element 'MarketDate' has invalid child element 'Date'. List of possible elements expected: 'DateFormat'."
Here's the relevant XML:
<MarketDate>
<MarketDateRole>01</MarketDateRole>
<Date>20150202</Date>
</MarketDate>
Here's what's in the DTD:
<!ELEMENT MarketDate (MarketDateRole,DateFormat,Date)>
So 'Date' is a valid child element. I am guessing it's not that Date is invalid but that DateFormat is missing. Am I correct with this assumption?

Big Query Pagination in vb.net

I am trying to page through BigQuery data with vb.net. I keep getting the same first page of data with my code. The way I understand, I need to set the pagetoken of the response to look at the next page.
With the following code, I only get the first page of data while never exiting my loop.
For the login I was setting the Oauthtoken of my queryrequest and that was getting the first page fine, but no attempt is made to page through the results that way.
I appreciate any lead in the right direction.
Dim DT As New DataTable
Dim ErrMessage As String = ""
Try
Dim INIT As New BigqueryService.Initializer
Dim scopes As IList(Of String) = New List(Of String)()
scopes.Add(BigqueryService.Scope.Bigquery)
Dim credential As UserCredential
Using stream As New FileStream("client_secrets.json", FileMode.Open, FileAccess.Read)
credential = GoogleWebAuthorizationBroker.AuthorizeAsync(GoogleClientSecrets.Load(stream).Secrets, scopes, "user", CancellationToken.None, New FileDataStore("BQ.App")).Result
End Using
INIT.HttpClientInitializer = credential
Dim service As New BigqueryService(INIT)
Dim j As JobsResource = service.Jobs
Dim req As New QueryRequest
req.Query = tQuery.Text
Dim QRequest As JobsResource.QueryRequest = j.Query(req, projectId)
QRequest.OauthToken = MyAccessToken
Dim JOBID As String = QRequest.Execute.JobReference.JobId
Dim DATA = QRequest.Execute
Dim schema = DATA.Schema
For Each col In schema.Fields
DT.Columns.Add(col.Name)
Next
Dim page_Tok = ""
Dim rr As GetQueryResultsResponse
While True
Try
rr.PageToken = page_Tok
Catch ex As Exception
'No Token Yet
End Try
rr = j.GetQueryResults(projectId, JOBID).Execute
page_Tok = rr.PageToken
If rr.JobComplete = True Then
If page_Tok = "" Then
Exit While
End If
End If
Dim resp2 = rr.Rows
For Each row In resp2
Dim DR As DataRow = DT.NewRow
For f = 0 To row.F.Count - 1
Dim field = row.F
DR(f) = row.F(f).V
Next
DT.Rows.Add(DR)
Next
End While
Catch ex As Exception
ErrMessage = ex.Message
End Try
I'm not a VB expert, but you're not setting the page token in the GetQueryResultsRequest, you're setting it in the response. See https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/csharp/latest/classGoogle_1_1Apis_1_1Bigquery_1_1v2_1_1JobsResource_1_1GetQueryResultsRequest.html
I think that this will work:
req = j.GetQueryResults(projectId, JOBID)
req.PageToken = page_tok
rr = req.Execute

Placement of constructor

For Some reason in the following segment in my for each statment its declaring the that the csv data cant be null even though as you see their when i create the csvData Object is its created within the loop is that the correct placement.
For Each thisDocument As String In documentList
filename = Path.GetFileName(thisDocument)
SiteId = filename.Substring(0, filename.IndexOf("_"))
Dim orderNumbers As String()
Dim csvData As New DataTable
Dim importUtils As New ImportController
filename = Path.GetFileName(thisDocument)
orderNumbers = filename.Split("_")
location = filename.Substring(0, 3)
importUtils = New ImportController(cfb.TargetFolder & filename)
csvData = importUtils.ConvertCsvToDatatable(True)
ImportDataTableToSql(cfb.TargetFolder & filename, csvData, cfb.StoreCompany)
Next
edits to show convertcsvtodatatable()
Public Function ConvertCsvToDatatable(ByVal ColumnNames As Boolean) As DataTable
Try
Dim dt As New DataTable
For Each columnName In GetColumnsfromCsv(ColumnNames, ",")
dt.Columns.Add(columnName)
Next
Dim fileReader As New StreamReader(FileName)
If ColumnNames Then
fileReader.ReadLine()
End If
Dim line As String = fileReader.ReadLine
While Not IsNothing(line)
line = line.Replace(Chr(34), "")
dt.Rows.Add(line.Split(","))
line = fileReader.ReadLine
End While
fileReader.Close()
Return dt
Catch ex As Exception
'log to file
End Try
Return Nothing
End Function