I am having issues connecting to the SoftLayer Product/Orders APIs.
Endpoints: Product_Packages is dropping the connection, Product_Place_Orders is returning invalid auth. Hardware and VirtualGuests are accepting the credentials and returning the information I expect.
Either:
Invalid credentials
Connection reset by peer
I am, however, able to use the same credentials to see what machines and vlans are in the account.
Looking into this more, curl works, another client I have works still, however the SoftLayer python client stopped working overnight.
Did the API change Auth mechanisms change last night?
import SoftLayer
client = SoftLayer.create_client_from_env() # or
client = SoftLayer.create_client_from_env(endpoint_url="https://api.softlayer.com/rest/v3/")
location_groups = client.call('Location_Group_Pricing', 'getAllObjects')
Output:
Unexpected error: <class 'SoftLayer.exceptions.TransportError'>
Traceback (most recent call last):
File "list_product_options.py", line 28, in <module>
location_groups = client.call('Location_Group_Pricing', 'getAllObjects')
File "/Users/tony/alchemy/ibm-krobots/sl-tools/slcli/lib/python2.7/site-packages/SoftLayer/API.py", line 263, in call
return self.transport(request)
File "/Users/tony/alchemy/ibm-krobots/sl-tools/slcli/lib/python2.7/site-packages/SoftLayer/transports.py", line 199, in __call__
raise exceptions.TransportError(0, str(ex))
SoftLayer.exceptions.TransportError: TransportError(0): ('Connection aborted.', error(54, 'Connection reset by peer'))
catching the exception shows the following:
Unexpected error: <class 'SoftLayer.exceptions.TransportError'>
A working curl for the same endpoint:
curl https://<username>:<api_key>#api.softlayer.com/rest/v3/SoftLayer_Location_Group_Pricing/getAllObjects.json
Also verified that the Golang client works.
https://github.com/softlayer/softlayer-go
Just to point out that I am experiencing the same issue and I believe there is another post from someone else also mentioning the same transport/connection forcibly closed issue.
Worked last night, not working today. I'm changing my api key and user_name to see if it gets fixed. Note I too have multiple SL accounts. I'm swapping to my primary account API credentials to see what happens.
Here is the error
Unable to read data from the transport connection: An existing connection was fo rcibly closed by the remote host.
System.Net.WebException: The underlying connection was closed: An unexpected err or occurred on a send. --->
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the
remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequestrequest)
at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at SoftlayerAPIV2.Softlayer.Account.SoftLayer_AccountService.getAdcLoadBalancers
After talking with SoftLayer support, there was a service issue which has now been resolved.
Future Mitigation:
More extensive testing for older clients and related systems will be implemented into our Change Management guidelines to prevent this type of issue in the future.
(I like that part :)
This was an issue in Softlayer side, please try again it should be fixed now.
Regards
We don't have any report about issues for that, and I'm able to use SoftLayer_Product_Order service, could you provide your request(s)? for further assistance
How often have you been getting this issue?
Did you have success with those request before?
Related
I'm connecting to a PostgreSQL database with the npgsql connector.
On average, with one out of every 50 000 queries, I get the following error. (This was while filling a dataset.)
This is the Exception Message:
The Connection is broken.
This is the Exception Inner Exception:
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. --->
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.FlushWrite()
at Npgsql.NpgsqlQuery.WriteToStream(Stream outputStream)
at Npgsql.NpgsqlReadyState.QueryEnum(NpgsqlConnector context, NpgsqlCommand command)
at Npgsql.NpgsqlCommand.GetReader(CommandBehavior cb)
Please let me know if I should rather post this in dba and if so I will repost.
Thanks.
We are using a multi server environment with 1 CMS and 2 CD configuration. The CORE and WEB of CMS are being shared by CD 2 only and CD 1 as its own web and core.
The issue is- log files of CD2 are capturing the below mentioned error:
ManagedPoolThread #14 00:00:05 INFO Job started:Index_Update_IndexName=sitecore_core_index
ManagedPoolThread #14 00:00:05 ERROR Exception
Exception: System.Reflection.TargetInvocationException
Message: Exception has been thrown by the target of an invocation.
Source: mscorlib
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Jobs.Job.ThreadEntry(Object state)
Nested Exception
Exception: System.InvalidOperationException
Message: Configuration
Source: Sitecore.ContentSearch.LuceneProvider
at Sitecore.ContentSearch.LuceneProvider.LuceneIndex.EnsureInitialized()
at Sitecore.ContentSearch.LuceneProvider.LuceneIndex.CreateUpdateContext()
at Sitecore.ContentSearch.LuceneProvider.LuceneIndex.PerformUpdate(IEnumerable`1 indexableUniqueIds, IndexingOptions indexingOptions)
As we researched, we found that this type of error gets captured when issue with Core database index update. Further this was also confirmed by crawling log and checking the configuration within the \App_Config\Include\Sitecore.ContentSearch.Lucene.Index.Core which has this job defined for Index update.
Crawling log entries getting captured
INFO [Index=sitecore_core_index] IntervalAsynchronousUpdateStrategy executing.
INFO [Index=sitecore_core_index] Event Queue is forced
As a next step we took following steps but problem still exist:
Deleted old Core indexes and rebuilt for CMS and CD2 both.
My Questions:
If we have same core for CMS and CD2 then do we require index rebuilt for both environment because as per my info the index get saved in disk.
Secondly we are facing caching issue for CD2 only, is this related to index update.
Thirdly the config details Sitecore.ContentSearch.Lucene.Index.Core are present in CMS and CD2 also so is this responsible for these exception as two processs are updating indexes for the same Core indexes.
Lastly we have a separate core and web for CD1 with replication enabled but we still see these error being captured in CD1 as well. We are seeing this errors from past few months and recently the frequency of errors has increased due to which it is eating a huge chunk of physical memory in CD2 and causing application higher response time.
I figured out what the problem in my environment was. Look in your log file and before you seeing that exeption, do you see the following:
ManagedPoolThread #11 10:07:24 INFO Loading Dictionary from database. Domain: 'Dictionary'. Language: 'en'.
ManagedPoolThread #11 10:07:24 ERROR Error saving {sitecore file path}\Website\temp\dictionary.dat.
Exception: System.IO.IOException
Message: The file '{sitecore file path}\Website\temp\dictionary.dat' already exists.
Source: mscorlib
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
at Sitecore.Globalization.Translate.Save()
If so, delete that and restart your site. This should fix the issue that you are seeing.
We have seen a noticeable uptick in problems related to RavenDB connections. We're using the IIS server connection method.
I've checked the index errors and I don't see anything listed.
This server is replicated with a MySQL server and there is one error related to replication in the log from today. Could the client stack trace errors actually be coming from the replicated server and not Raven?
EDIT
The client in this case is a single job that runs a few hundred times a day...succeeding many of those times but increasing in failures due to these errors.
Sql Replication failure to replication
Here are the partial stack trace errors from the client log:
[WebException: Unable to connect to the remote server]
System.Net.HttpWebRequest.GetRequestStream(TransportContext& context):367
System.Net.HttpWebRequest.GetRequestStream():3
Raven.Abstractions.Connection.HttpRequestHelper.WriteDataToRequest(HttpWebRequest req, String data, Boolean disableCompression):7
Raven.Client.Connection.ServerClient.DirectBatch(IEnumerable`1 commandDatas, String operationUrl):171
Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result, Boolean& wasTimeout):35
Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation):169
Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation):33
Raven.Client.Document.DocumentSession.SaveChanges():65
and
[WebException: Unable to connect to the remote server]
System.Net.HttpWebRequest.GetResponse():570
Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse):45
Raven.Client.Connection.HttpJsonRequest.ReadResponseJson():206
Raven.Client.Connection.ServerClient.DirectGet(String[] ids, String operationUrl, String[] includes, String transformer, Dictionary`2 queryInputs, Boolean metadataOnly):631
Raven.Client.Connection.ServerClient+<>c__DisplayClass77.<Get>b__76(String u):51
Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, String operationUrl, Boolean avoidThrowing, T& result, Boolean& wasTimeout):35
Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation):169
Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation):33
Raven.Client.Document.HiLoKeyGenerator.GetDocument(IDatabaseCommands databaseCommands):41
Raven.Client.Document.HiLoKeyGenerator.GetNextRange(IDatabaseCommands databaseCommands):109
Raven.Client.Document.HiLoKeyGenerator.NextId(IDatabaseCommands commands):58
Raven.Client.Document.HiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention convention, Object entity):9
Raven.Client.Document.MultiTypeHiLoKeyGenerator.GenerateDocumentKey(IDatabaseCommands databaseCommands, DocumentConvention conventions, Object entity):174
Raven.Client.Document.DocumentStore+<>c__DisplayClass4.<Initialize>b__2(String dbName, IDatabaseCommands databaseCommands, Object entity):20
Raven.Client.Document.DocumentConvention.GenerateDocumentKey(String dbName, IDatabaseCommands databaseCommands, Object entity):164
Raven.Client.Document.GenerateEntityIdOnTheClient.GenerateDocumentKeyForStorage(Object entity):46
Raven.Client.Document.InMemoryDocumentSessionOperations.StoreInternal(Object entity, Etag etag, String id, Boolean forceConcurrencyCheck):79
Raven.Client.Document.InMemoryDocumentSessionOperations.Store(Object entity):23
The error says the client cannot connect to the the server. Maybe the client is offline. Or the server is. Or a firewall is in the way. Or maybe the server is there but blows up before returning a response to the client. If the issue is intermittent, and users arent complaining, it's probably just a connectivity thing that you can ignore. If users are complaining, you should look in server-side logs.
I am getting some random disconnects from Booksleeve when deployed in the production environment. I have run tests on my machine and I can not seem to reproduce the errors. I am not sure what to check so I decided to post here.
My set-up:
2 x Redis 2.6.16 (Debian Linux)
2 x Redis sentinel (running on the same machines as redis)
4 x Windows Server 2008 R2
Application connection handling
My app has one static connection that is opened using ConnectionUtils. I also have a timer (System.Timers.Timer) that runs every connection.Ping every 5'' to check whether the connection is still open. If I get an exception for the Ping I log it, close and dispose the connection and replace it with a new one using ConnectionUtils again.
I am also logging Task.UnobservedExceptions
Here is what I am getting in my logs:
A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at BookSleeve.RedisMessage.WriteUnified(Stream stream, Byte[] value) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 271
at BookSleeve.RedisMessage.RedisMessageMulti.Write(Stream stream) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 564
at BookSleeve.RedisConnectionBase.WriteMessage(Int32& db, RedisMessage next, IList`1 queued) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1597
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at BookSleeve.RedisMessage.WriteUnified(Stream stream, Byte[] value) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 271
at BookSleeve.RedisMessage.RedisMessageMulti.Write(Stream stream) in d:\Dev\BookSleeve\BookSleeve\RedisCommand.cs:line 564
at BookSleeve.RedisConnectionBase.WriteMessage(Int32& db, RedisMessage next, IList`1 queued) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1597
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at System.Threading.Tasks.Task.Execute()
And this is my connection monitor reporting on the error in the ping:
Redis connection is broken: System.InvalidOperationException: The connection has been closed (Error); no new messages can be delivered; the last command was sent 1453ms ago
at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
at BookSleeve.RedisConnectionBase.ExecuteInt64(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1688
at BookSleeve.RedisConnectionBase.PingImpl(Boolean queueJump, Boolean duringInit, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 129
at Atcom.Data.Caching.RedisConnectionContainer.CheckConnections(Object sender, EventArgs e) in C:\Projects\Atcom.Data.Caching.Redis\Atcom.Data.Caching.Redis\RedisConnectionContainer.cs:line 66
I have tried connecting through both the Sentinels and directly to Redis but I am getting the same thing.
I have asked my network administrator to check but he says there is no problem (and I cant really argue with what I dont know oO)
I have reviewed Redis settings and documentation and as far as I can tell clients that do not use monitor etc. should not get disconnected by redis. Idle time disconnect is also disabled on redis.
Could sentinel (since its beta) be the culprit?
Any help troubleshooting the problem would be GREATLY appreciated.
I have a webservice that works perfectly from localhost but when I hosted it on the testing server I get an error on one of the methods. I am using wsHttp Binding. everything else works fine on the client side except for this method. This is the error that I am getting
client error:
System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied.
Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Server error:
namespace.Service Error: 10001 : Error occurred in methodname().
System.Security.SecurityException: The source was not found, but some or all event logs could not be searched. Inaccessible logs: Security.
at System.Diagnostics.EventLog.FindSourceRegistration(String source, String machineName, Boolean readOnly)
at System.Diagnostics.EventLog.SourceExists(String source, String machineName)
at System.Diagnostics.EventLog.VerifyAndCreateSource(String sourceName, String currentMachineName)
at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Byte[] data, Object[] values)
at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Object[] values)
at System.Diagnostics.EventLogTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType severity, Int32 id, String format, Object[] args)
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String format, Object[] args)
at AutoWatch.Entity.WcfService.TrackingService.UpdateIncidentStatusHistory(Int64 incidentId, String status, String username, String comment, Boolean SuspectFaultyUnit) in C:\..servicename.cs:line 566
at AutoWatch.Entity.WcfService.TrackingService.GetNewIncidentMessage(String username) in C:\..servicename.cs:line 444
The Zone of the assembly that failed was:
MyComputer
I added in the error I am getting on the server.
Is it possible I am getting this error because the service cannot write to the event log?
Please help.
Ensure that account running your process where the service is hosted has access rights to the database. For example in case of IIS the account running the application pool where the service is hosted must have login to database server and it must have permissions to do all necessary operations in your database.
Edit:
The server stack trace looks quite straightforward. You have a problem with writing to Windows Event Log! It cannot find source you requested and it doesn't have permission to create it.
I got the same error when I tried to use the following statement in a Web Method of a WCF Service:
string myTypeName = typeof(ErrorHandlerBehavior).AssemblyQualifiedName;
where ErrorHandlerBehavior derives from BehaviorExtensionElement.
The line works well when I test the service on my localhost. On the web, the call of the containing method raises the Exception "Access is denied".
In this case, I guess that the cause was that my IP does not allow the creation of a BehaviourExtensionElement in a Partially Trusted environment (my service is in a shared hosting environment).
Finally, I succeded following the first mechanism described at https://learn.microsoft.com/en-us/dotnet/framework/wcf/extending/configuring-and-extending-the-runtime-with-behaviors about Service Behaviors:
"Using an attribute on the service class. When a ServiceHost is constructed, the ServiceHost implementation uses reflection to discover the set of attributes on the type of the service. If any of those attributes are implementations of IServiceBehavior, they are added to the behaviors collection on ServiceDescription. This allows those behaviors to participate in the construction of the service run time."
I just modified the derivation (no other modification required):
public class ErrorServiceBehavior : Attribute, IServiceBehavior
{ ... }
and use the class as Attribute of my service:
[ErrorServiceBehavior()]
public partial class MyService : IMyService
{...}
No other modification required. Check the original sample at How do I create a global exception handler for a WCF Services?.