Solr - Error when posting an "Add" to the server - lucene

I'm Posting the following to the Solr Server:
<add>
<doc>
<field name="uniqueid">5453543</field>
<field name="modifieddate">2008-12-03T15:49:00Z</field>
<field name="title">My Record</field>
<field name="description">Descirption
Details
</field>
<field name="startdate">2009-01-21T15:26:05.680Z</field>
<field name="enddate">2009-01-21T15:26:05.697Z</field>
<field name="Telephone_number">11111 111 111(text phone)</field>
<field name="Telephone_number">11111 111 111</field>
<field name="Mobile_number">1111111111</field>
</doc>
</add>
I'm using SolrNet to send the documents here's an extract from the code (s is the above xml):
public string Post(string relativeUrl, string s)
{
var u = new UriBuilder(serverURL);
u.Path += relativeUrl;
var request = httpWebRequestFactory.Create(u.Uri);
request.Method = HttpWebRequestMethod.POST;
request.KeepAlive = false;
if (Timeout > 0)
request.Timeout = Timeout;
request.ContentType = "text/xml; charset=utf-8";
request.ContentLength = s.Length;
request.ProtocolVersion = HttpVersion.Version10;
try
{
using (var postParams = request.GetRequestStream())
{
postParams.Write(xmlEncoding.GetBytes(s), 0, s.Length);
using (var response = request.GetResponse())
{
using (var rStream = response.GetResponseStream())
{
string r = xmlEncoding.GetString(ReadFully(rStream));
//Console.WriteLine(r);
return r;
}
}
}
}
catch (WebException e)
{
throw new SolrConnectionException(e);
}
}
When it gets to request.GetResponse it failed with this error:
base
{System.InvalidOperationException} =
{"The remote server returned an error:
(500) Internal Server Error."}
When i look on the server in the Logs for apache it gives the following reason:
Unexpected end of input block in end
Here's the full stack trace:
Sep 17, 2009 10:13:53 AM
org.apache.solr.common.SolrException
log SEVERE:
com.ctc.wstx.exc.WstxEOFException:
Unexpected end of input block in end
tag at [row,col {unknown-source}]:
[26,1266] at
com.ctc.wstx.sr.StreamScanner.throwUnexpectedEOB(StreamScanner.java:700)
at
com.ctc.wstx.sr.StreamScanner.loadMoreFromCurrent(StreamScanner.java:1054)
at
com.ctc.wstx.sr.StreamScanner.getNextCharFromCurrent(StreamScanner.java:811)
at
com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3211)
at
com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2832)
at
com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
at
org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:148)
at
org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at
org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:859)
at
org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:574)
at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1527)
at
java.lang.Thread.run(Thread.java:619)
Please note the Solr server is running on the following system:
Microsoft Windows Server 2003 R2
Apache Tomcat 6
Finally here's my question:
The Xml i'm sending looks ok to me.. Does anyone have an ideas as to why Solr is throwing this exception?
Thanks
Dave
Edit Answer is as follows:
public string Post(string relativeUrl, string s)
{
var u = new UriBuilder(serverURL);
u.Path += relativeUrl;
var request = httpWebRequestFactory.Create(u.Uri);
request.Method = HttpWebRequestMethod.POST;
request.KeepAlive = false;
if (Timeout > 0)
request.Timeout = Timeout;
request.ContentType = "text/xml; charset=utf-8";
request.ProtocolVersion = HttpVersion.Version10;
try
{
// Set the Content length after the size of the byte array has been calculated.
byte[] data = xmlEncoding.GetBytes(s);
request.ContentLength = s.Length;
using (var postParams = request.GetRequestStream())
{
postParams.Write(data, 0, data.Length);
using (var response = request.GetResponse())
{
using (var rStream = response.GetResponseStream())
{
string r = xmlEncoding.GetString(ReadFully(rStream));
//Console.WriteLine(r);
return r;
}
}
}
}
catch (WebException e)
{
throw new SolrConnectionException(e);
}
}

I'm not much familiar with .Net or Solr or .Net port of Solr. But, here is my guess.
postParams.Write(xmlEncoding.GetBytes(s), 0, s.Length);
There are two possible errors.
When you are getting bytes from String, you should specify the encoding. It might be the case that the default encoding is different from UTF8, which you have set in content type header in response.
The third parameter in Write() probabably refers to the length of byte array which you got from GetBytes(). The byte array could be longer than the length of string.

Related

Storing An Image In SQL Server

I need to create a procedure in SQL server that takes a web URL of an image and converts it to VARBINARY, and after that: store in a column called "personqr_Image" in table "tblPersons".
I created a procedure "getPersonQrCode" that returns a URL of a unique QR code (450x450 image), and using that URL I need to convert it to VARBINARY data type in order to store it in my SQL DB.
Unfortunately I haven't really found a solution online, maybe because I am not very familiar with the subject.
You can't do this purely in TSQL, as it doesn't have any functions for browsing the web and handling http requests and responses. If you have to do this IN SQL Server, you'll need to write a CLR procedure.
Here is a CLR function that will allow you to submit HTTP requests
public class RestClient
{
[SqlFunction(DataAccess = DataAccessKind.Read)]
public static string Submit(string url, string data, string contentType, string
method = "POST",
string httpHeaderCredentials = "")
{
try
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls |
SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
var request = (HttpWebRequest) WebRequest.Create(url);
//Add header credentials if required
if (!string.IsNullOrEmpty(httpHeaderCredentials))
{
request.Headers.Add("Authorization: " + httpHeaderCredentials);
}
request.ContentType = contentType;
request.Method = method;
if (request.Method == "PATCH")
{
//http://stackoverflow.com/questions/31043195/rest-api-patch-request
request.ServicePoint.Expect100Continue = false;
}
if (method == "POST" || method == "PATCH")
{
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
{
streamWriter.Write(data);
streamWriter.Flush();
streamWriter.Close();
}
}
var httpResponse = request.GetResponse();
using (var responseStream = httpResponse.GetResponseStream())
{
if (responseStream != null)
{
using (var reader = new StreamReader(responseStream))
{
return reader.ReadToEnd().Replace("\n", string.Empty);
}
}
}
}
catch (Exception ex)
{
if (SqlContext.Pipe != null)
{
SqlContext.Pipe.Send(ex.Message);
}
}
return "";
}

Error Code 551 while connecting a FTP server through .NET code

I am using the below c# code to connect the FTP server. Sometimes while sending the files it is getting the Error Code 551 with Description: Exception caught in sending FTP: The remote server returned an error: (551) Page type unknown.
I don't why this is happening.
Can anyone tell me what the issue is?
private bool sendFTP(string sDestPath, string sFileName, string sUserName, string sPassword, string sDomain, bool isProxyUsed, string sProxy, int nProxyPort, byte[] sData)
{
try
{
NetworkCredential nCred;
if (!sDomain.Equals(String.Empty))
{
nCred = new System.Net.NetworkCredential(sUserName, sPassword, sDomain);
}
else
{
nCred = new System.Net.NetworkCredential(sUserName, sPassword);
}
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(sDestPath + "//" + sFileName);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = nCred;
request.Proxy = new WebProxy();
if (isProxyUsed)
{
WebProxy p = new WebProxy(sProxy, nProxyPort);
p.Credentials = nCred;
WebRequest.DefaultWebProxy = p;
}
request.ContentLength = sData.Length;
Stream reqStream = request.GetRequestStream();
reqStream.Write(sData, 0, sData.Length);
reqStream.Close();
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
return true;
}
catch (Exception objEx)
{
// Error
EventLog.WriteEntry("STCCommon", "Exception Caught:" + objEx.Message, EventLogEntryType.Error);
throw new Exception("Exception caught in sending FTP: " + objEx.Message);
}
}

Why am I getting, "Unable to connect to the remote server"?

This err msg is the next one I get after resolving “NotSupportedException” as noted here
I don't even reach the break point in the server code (set on the first line of the method that should be getting called).
This is the relevant server code:
[Route("api/PlatypusItems/PostArgsAndXMLFileAsStr")]
public async void PostArgsAndXMLFileAsStr([FromBody] string stringifiedXML, string serialNum, string siteNum)
{
string beginningInvoiceNum = string.Empty; // <= Breakpoint on this line
string endingInvoiceNum = string.Empty;
XDocument doc = XDocument.Load(await Request.Content.ReadAsStreamAsync());
. . .
And the client (handheld, Compact Framework) code:
private void menuItem4_Click(object sender, EventArgs e)
{
GetAndSendXMLFiles("LocateNLaunch"); // There is a "LocateNLaunch.xml" file
}
private void GetAndSendXMLFiles(string fileType)
{
string serNum = User.getSerialNo();
string siteNum = User.getSiteNo();
if (serNum.Length == 0)
{
serNum = "8675309";
}
if (siteNum.Length == 0)
{
siteNum = "03";
}
string uri = string.Format("http://localhost:28642/api/PlatypusItems/PostArgsAndXMLFileAsStr?serialNum={0}&siteNum={1}", serNum, siteNum);
List<String> XMLFiles = HHSUtils.GetXMLFiles(fileType, #"\");
MessageBox.Show(XMLFiles.Count.ToString());
foreach (string fullXMLFilePath in XMLFiles)
{
MessageBox.Show(fullXMLFilePath);
RESTfulMethods.SendXMLFile(fullXMLFilePath, uri, 500);
}
}
public static string SendXMLFile(string xmlFilepath, string uri, int timeout) // timeout should be 500
{
MessageBox.Show(string.Format("In SendXMLFile() - xmlFilepath == {0}", xmlFilepath));
MessageBox.Show(string.Format("In SendXMLFile() - uri == {0}", uri));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
request.Method = "POST";
StringBuilder sb = new StringBuilder();
using (StreamReader sr = new StreamReader(xmlFilepath))
{
String line;
while ((line = sr.ReadLine()) != null)
{
sb.AppendLine(line);
}
byte[] postBytes = Encoding.UTF8.GetBytes(sb.ToString());
if (timeout < 0)
{
request.ReadWriteTimeout = timeout;
request.Timeout = timeout;
}
request.ContentLength = postBytes.Length;
request.KeepAlive = false;
request.ContentType = "application/xml";
try
{
Stream requestStream = request.GetRequestStream();
requestStream.Write(postBytes, 0, postBytes.Length);
requestStream.Close();
using (var response = (HttpWebResponse)request.GetResponse())
{
return response.ToString();
}
}
catch (Exception ex)
{
MessageBox.Show("SendXMLFile exception " + ex.Message);
request.Abort();
return string.Empty;
}
}
}
Running this code, I see from the client the following "debug strings":
0) "1" (from MessageBox.Show(XMLFiles.Count.ToString());)
1) "\Program Files\LocateNLaunch\LocateNLaunch.xml" (from MessageBox.Show(fullXMLFilePath);)
2) "In SendXMLFile() - xmlFilePath == \Program Files\LocateNLaunch\LocateNLaunch.xml" (from MessageBox.Show(string.Format("In SendXMLFile() - xmlFilepath == {0}", xmlFilepath));)
3) "In SendXMLFile() - uri == http://localhost:28642/api/PlatypusItems/PostArgsAndXMLFileAsStr?serialNum=8675309&siteNum=03" (from MessageBox.Show(string.Format("In SendXMLFile() - uri == {0}", uri));)
- and then this one from somewhere:
4) "SendXMLFile exception Unable to connect to the remote server"...
So what could be causing this inability to connect?
UPDATE
The same thing ("Unable to Connect to the Remote Server") happens with this code (different operation, but also from the WindowsCE/Compact Framework/handheld app that tries to connect to the Web API server app):
private void menuItem3_Click(object sender, EventArgs e)
{
string serNum = User.getSerialNo();
if (serNum.Length == 0)
{
serNum = "8675309";
}
string clientVer =
HHSUtils.GetFileVersion(#"\Application\sscs\vsd_setup.dll");
if (clientVer.Contains("Win32Exception"))
{
clientVer = "0.0.0.0";
}
MessageBox.Show(string.Format("After call to GetFileVersion(), serial num == {0};
clientVer == {1}", serNum, clientVer));
string uri =
string.Format("http://localhost:28642/api/FileTransfer/GetHHSetupUpdate?
serialNum={0}&clientVersion={1}", serNum, clientVer);
RESTfulMethods.DownloadNewerVersionOfHHSetup(uri);
}
public static void DownloadNewerVersionOfHHSetup(string uri)
{
string dateElements = DateTime.Now.ToString("yyyyMMddHHmmssfff",
CultureInfo.InvariantCulture);
var outputFileName = string.Format("HHSetup_{0}.exe", dateElements);
try
{
var webRequest = (HttpWebRequest)WebRequest.Create(uri);
var webResponse = (HttpWebResponse)webRequest.GetResponse();
string statusCode = webResponse.StatusCode.ToString();
if (statusCode == "NoContent")
{
MessageBox.Show("You already have the newest available version.");
}
else
{
var responseStream = webResponse.GetResponseStream();
using (Stream file = File.Create(outputFileName))
{
CopyStream(responseStream, file);
MessageBox.Show(string.Format("New version downloaded to {0}",
outputFileName));
}
}
}
catch (WebException webex)
{
MessageBox.Show("DownloadNewerVersionOfHHSetup: " + webex.Message);
}
}
// I see the "After call to GetFileVersion()" message in menuItem3_Click() handler, but then "DownloadNewerVersionOfHHSetup: Unable to Connect to the Remote Server" in DownloadNewerVersionOfHHSetup()
And yes, the server app is running.
UPDATE 2
Here is the code that I tested prior to "dumbing it down" (retrofitting it, making it as similar as possible to this working test code, yet that may not be saying much) for Compact Framework:
Client code:
DownloadTheFile(textBoxFinalURI.Text); // with textBoxFinalURI.Text being
"http://localhost:28642/api/FileTransfer/GetUpdatedHHSetup?
serialNum=8675309&clientVersion=1.3.3.3" and the file on the server being
version 1.4.0.15
private void DownloadTheFile(string uri)
{
var outputFileName = "Whatever.exe";
try
{
var webRequest = (HttpWebRequest)WebRequest.Create(uri);
var webResponse = (HttpWebResponse)webRequest.GetResponse();
string statusCode = webResponse.StatusCode.ToString();
if (statusCode == "NoContent")
{
MessageBox.Show("You already have the newest available version.");
}
else
{
var responseStream = webResponse.GetResponseStream();
using (Stream file = File.Create(outputFileName))
{
CopyStream(responseStream, file);
MessageBox.Show(string.Format("New version downloaded to {0}",
outputFileName));
}
}
}
catch (WebException webex)
{
MessageBox.Show(webex.Message);
}
}
Server code:
public HttpResponseMessage GetHHSetupUpdate(string serialNum, string clientVersion)
{
HttpResponseMessage result;
string filePath = GetAvailableUpdateForCustomer(serialNum);
FileVersionInfo currentVersion = FileVersionInfo.GetVersionInfo(filePath);
if (!ServerFileIsNewer(clientVersion, currentVersion))
{
result = new HttpResponseMessage(HttpStatusCode.NoContent);
}
else
{
result = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new FileStream(filePath, FileMode.Open);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream");
}
return result;
}
private string GetAvailableUpdateForCustomer(string serialNum)
{
if (serialNum == "8675309")
{
return HostingEnvironment.MapPath(#"~\App_Data\HHSetup.exe");
}
else
{
return HostingEnvironment.MapPath(#"~\App_Data\HDP.exe");
}
}
// clientFileVersion is expected to be something like "1.4.0.15"
private bool ServerFileIsNewer(string clientFileVersion, FileVersionInfo serverFile)
{
Version client = new Version(clientFileVersion);
Version server = new Version(string.Format("{0}.{1}.{2}.{3}",
serverFile.FileMajorPart, serverFile.FileMinorPart,
serverFile.FileBuildPart, serverFile.FilePrivatePart));
return server > client;
}
... This code works fine (server code is the same; the client code has been "retrofied")
I can't use the code as-is because of the limitations of Compact Framework / Windows CE. As the title of this post makes clear, I'm not even able to connect to the server from there yet. Is it possible? If so, what needs to change in my client code (not the client code in Update 2, which works in newer versions of .NET, but the client code shown prior to there)?
It's a similar story with the other method that is also returning "Unable to connect to the remote server" - it works fine in "modern" code running in a test app, but once it's retrofitted (better word than refactored when "dumbing down" to Compact Frameworkerize the code).
UPDATE 3
I tried to get more info from the err msg with the code below (old line commented out), but this "rewards" me instead with a NullReferenceException:
catch (WebException webex)
{
//MessageBox.Show("DownloadNewerVersionOfHHSetup: " + webex.Message);
string msg = webex.Message;
string innerEx = webex.InnerException.ToString();
string resp = webex.Response.ToString();
string stackTrace = webex.StackTrace;
string status = webex.Status.ToString();
MessageBox.Show(
string.Format("Message: {0}; Inner Exception: {1}; Response: {2}; Stack Trace: {3}; Status: {4}", msg, innerEx, resp, stackTrace, status));
}
UPDATE 4
As I continued to get NREs, I commented out each subsequent line, one-by-one, until I now have this that runs:
//string innerEx = webex.InnerException.ToString();
//string resp = webex.Response.ToString();
//string stackTrace = webex.StackTrace;
string status = webex.Status.ToString();
MessageBox.Show(
//string.Format("Message: {0}; Inner Exception: {1}; Response: {2}; Stack Trace: {3}; Status: {4}", msg, innerEx, resp, stackTrace, status));
//string.Format("Message: {0}; Response: {1}; Stack Trace: {2}; Status: {3}", msg, resp, stackTrace, status));
//string.Format("Message: {0}; Stack Trace: {1}; Status: {2}", msg, stackTrace, status));
string.Format("Message: {0}; Status: {1}", msg, status));
...but all I get from it is Status of "ConnectFailure" (I already knew that).
UPDATE 5
This runs without an NRE:
string msg = webex.Message;
string innerEx = webex.InnerException.ToString();
string status = webex.Status.ToString();
MessageBox.Show(string.Format("Message: {0}; Status: {1}; inner Ex: {2}", msg, status, innerEx));
And this is what I see:
So why would the server actively refuse the connection?
BTW, ASAP I'm going to bountify this question, or will bountify the answerer after the fact*, with a bounty that would make even Long John Silver and Perro-Negro's eyes glimmer and gleam (cared they for geekCoin, that is).
For facts leading to the arrest and eviction of this bug.
PSYCHE! I changed my mind/there's been a mutiny on the bounty => the bountification will happen here instead.
UPDATE 6
This also (using the "raw" IP Address of the server machine) gives me an NRE:
string uri = string.Format("http://192.168.125.50:28642/api/FileTransfer/GetHHSetupUpdate?serialNum={0}&clientVersion={1}", serNum, clientVer);
...as does using the "friendly name" ("Platypus") of the machine in place of the IP Address.
The large problem I see here is the fact that you have localhost as your address. That's absolutely wrong. localhost means, effectively, "on the same machine as I am running" so unless you've somehow managed to get a async .NET 4.0 web service to run on your Windows CE device and your server code is running there, then this is most certainly not what you want.
If you're running on an emulator, it's still wrong. The emulator is, for all intents and purposes, a separate machine.
You must use the address of the server/PC where that web service is running. It must be a routable address, meaning if you're connected over USB then it's probably ppp_peer and not an IP address (well it resolves to a private address, but the name is easier to remember).

2 exceptions when trying to make async HttpWebRequest

I am writing an MVC Web API the make async HttpWebRequest calls. I am getting 2 different exceptions. Below is the method I am using.
The 1st exception is: "This stream does not support seek operations." and it is happening on the responseStream.
The 2nd exception is: "timeouts are not supported on this stream" and that is happening on the MemoryStream content.
What am I doing wrong? I have been Googling but not really finding any solution.
Thanks,
Rhonda
private async Task GetHtmlContentAsync(string requestUri, string userAgent, string referrer, bool keepAlive, TimeSpan timeout, bool forceTimeoutWhileReading, string proxy, string requestMethod, string type)
{
//string to hold Response
string output = null;
//create request object
var request = (HttpWebRequest)WebRequest.Create(requestUri);
var content = new MemoryStream();
request.Method = requestMethod;
request.KeepAlive = keepAlive;
request.Headers.Set("Pragma", "no-cache");
request.Timeout = (Int32)timeout.TotalMilliseconds;
request.ReadWriteTimeout = (Int32)timeout.TotalMilliseconds;
request.Referer = referrer;
request.Proxy = new WebProxy(proxy);
request.UserAgent = userAgent;
try
{
using (WebResponse response = await request.GetResponseAsync().ConfigureAwait(false))
{
using (Stream responseStream = response.GetResponseStream())
{
if (responseStream != null)
{
await responseStream.CopyToAsync(content);
}
}
var sr = new StreamReader(content);
output = sr.ReadToEnd();
sr.Close();
}
}
catch (Exception ex)
{
output = string.Empty;
var message = ("The API caused an exception in the " + type + ".\r\n " + requestUri + "\r\n" + ex);
Logger.Write(message);
}
return output;
}
I fixed the issue by adding
content.Position = 0
before new StreamReader line. Now I just need to get it work with GZip compression.
Rhonda

forbidden:403 error while using wcf restful service

I am trying to consume wcf restful service. The code is as follows:
private static string SendRequest(string uri, string method, string contentType, string body)
{
string responseBody = null;
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Method = method;
if (!String.IsNullOrEmpty(contentType))
{
req.ContentType = contentType;
}
if (body != null)
{
byte[] bodyBytes = Encoding.UTF8.GetBytes(body);
req.GetRequestStream().Write(bodyBytes, 0, bodyBytes.Length);
req.GetRequestStream().Close();
}
req.Accept = "*/*";
HttpWebResponse resp;
resp = (HttpWebResponse)req.GetResponse();
return responseBody;
}
Now the issue is, sometimes it works fine and sometimes i get the error
"the remote server returned an error 403 forbidden."
I cannot figure out why it fails. Any idea???