I have a table in the default spark pool which i need to load into the dedicated sql pool in the azure synapse. Below are the codes I implemented, however it is not loading.
%%pyspark
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
new_df = spark.createDataFrame(segmentation_output)
new_df.write.mode("overwrite").saveAsTable("default.segmentation_output")
%%pyspark
new_df.createOrReplaceTempView("pysparkdftemptable")
%%spark
val scala_df = spark.sqlContext.sql ("select * from pysparkdftemptable")
scala_df.write.synapsesql("eana.bi.xim_CustomerSegment", Constants.INTERNAL)
Error : StructuredStream-spark package version: 2.4.5-1.3.1
StructuredStream-spark package version: 2.4.5-1.3.1
StructuredStream-spark package version: 2.4.5-1.3.1
com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host eec8e890e9d5--0.tr624.northeurope1-a.worker.database.windows.net (redirected from emna-dv-ibsanalytics-wco-id-euno-sqs.database.windows.net), port 11030 has failed. Error: "connect timed out. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".
Related
We have a setup wherein, one ignite server node serves 15 to 20 thick client nodes and 40 to 50 thin client nodes, thin client connection is singlton,
In operation, some times we get below error,
org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable [sock=Socket[addr=hostnm19.hostx.com/10.13.10.19,port=30519,localport=57552]]
On the Server node, we are inserting data inside a third party store using CacheStoreAdapters
Don't know where it goes wrong since out of 100 operations one operation fails with the above error.
Also, let me know what can we do for this failure handling.
Apache Ignite version: 2.8
Edits: (Code Snippet)
ClientConfiguration cfg = new ClientConfiguration()
.setAddresses("host:port");
IgniteClient client = Ignition.startClient(cfg); // this client is singleton
client.getOrCreateCache("ABC_CACHE").put(key, val);
StatckTrace:
org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable [sock=Socket[addr=hostnm19.hostx.com/10.13.10.19,port=30519,localport=57552]]
at org.apache.ignite.internal.client.thin.TcpClientChannel.handleIOError(TcpClientChannel.java:499)
at org.apache.ignite.internal.client.thin.TcpClientChannel.handleIOError(TcpClientChannel.java:491)
at org.apache.ignite.internal.client.thin.TcpClientChannel.access$100(TcpClientChannel.java:92)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.read(TcpClientChannel.java:538)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.readInt(TcpClientChannel.java:572)
at org.apache.ignite.internal.client.thin.TcpClientChannel.processNextResponse(TcpClientChannel.java:272)
at org.apache.ignite.internal.client.thin.TcpClientChannel.receive(TcpClientChannel.java:234)
at org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:171)
at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:160)
at org.apache.ignite.internal.client.thin.ReliableChannel.request(ReliableChannel.java:187)
at org.apache.ignite.internal.client.thin.TcpIgniteClient.getOrCreateCache(TcpIgniteClient.java:114)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.read(TcpClientChannel.java:535)
... 36 more
You probably have network or NAT configured which will reset connections when not used, or even sporadically.
In this case, you will have to reconnect.
Another option, are you sure you are connecting to thin client port and not some other port?
I want to connect my Airflow and Microsoft SQL Server. I configured my connection under 'connections' bar in 'Admin' box as mentioned in the following link:
http://airflow.apache.org/howto/manage-connections.html
But when I run my Dag task that is related to SQL server immedatly fails by following error:
[2019-03-28 16:16:07,439] {models.py:1788} ERROR - (18456, "Login failed for user 'XXXX'.DB-Lib error message 20018, severity 14:\nGeneral SQL Server error: Check messages from the SQL Server\nDB-Lib error message 20002, severity 9:\nAdaptive Server connection failed (***.***.***.28:1433)\n")
My code from DAG for Micrososft Sql Connection is following:
sql_command = """
select * from [sys].[tables]
"""
t3 = MsSqlOperator( task_id = 'run_test_proc',
mssql_conn_id = 'FIConnection',
sql = sql_command,
dag = dag)
I verified ip address and port number kind of configuration things by establishing connection through pymssql library from my local computer. Test code is following:
pymssql.connect(server="***.***.***.28:1433",
user="XXXX",
password="XXXXXX"
) as conn:
df = pd.read_sql("SELECT * FROM [sys].[tables]", conn)
print(df)
Could you please share if you have experienced this issue?
By the way I am using VirtualBox in Ubuntu 16.04 LTS
I had the same problem because freetds-dev was missing on linux:
apt-get install freetds-dev
I'm using dcm4chee docker stack with ldap and postgreSQL and have a floating error:
ldap:389; socket closed; remaining name 'cn=Devices,cn=DICOM Configuration,dc=mdw,dc=io'
2018-07-13 06:30:42,089 INFO [org.dcm4che3.conf.ldap.ReconnectDirContext] (Thread-0 (ActiveMQ-client-global-threads)) Connection to ldap://ldap:389 broken - reconnect
All three services are running on the same host. What can I do to avoid that error?
Disable the firewall. If firewall was enabled add an exception for all 3 services.
I have installed SQL Server 2012 Express on a Windows Server 2008 Machine. The name of the machine is "THEMACHINE". While installation, I set SQL Server 2012 as "Default Instance".
Now I set my connection string like this :
<add name="BooksContext" providerName="System.Data.SqlClient" connectionString="Data Source=THEMACHINE;Initial Catalog=newdb; User ID=sa; Password=123456"/>
But I get this exception :
{"A network-related or instance-specific error occurred while
establishing a connection to SQL Server. The server was not found or
was not accessible. Verify that the instance name is correct and that
SQL Server is configured to allow remote connections. (provider: Named
Pipes Provider, error: 40 - Could not open a connection to SQL
Server)"}
I checked that database is running. Can you tell me what I'm doing wrong?Thanks.
Reference: Creating and Configuring Universal Data Link (.udl) Files
Do following to Test connection and generate connection string,
1) Create query.udl file in the desktop
2) Double click the file to open and do the following
3)Open the Query.udl in notepad, you will find the connectionstring.
Change Data Source=THEMACHINE to Data Source=THEMACHINE\SQLEXPRESS. Also check that the database service is running.
I'm getting an error when running this query in SQL Server 2008:
EXEC master..xp_cmdshell'bcp
"select Row_ID, Service_Code,Skill from Account_Service"
queryout "d:\text.txt" -c -T -x'
Error is:
SQLState = 08001, NativeError = 2
Error = [Microsoft][SQL Server Native Client 10.0]Named Pipes Provider: Could not open a connection to SQL Server [2].
SQLState = 08001, NativeError = 2
Error = [Microsoft][SQL Server Native Client 10.0]A network-related or instance-specific error has occurred while
establishing a connection to SQL Server. Server is not found or not
accessible. Check if instance name is correct and if SQL Server is
configured to allow remote connections. For more information see
SQL Server Books Online.
SQLState = S1T00, NativeError = 0
Error = [Microsoft][SQL Server Native Client 10.0]
Login timeout expired
NOTE: Shared Memory, Named Pipes, TCP/IP are all enabled in SQL Server Configuration Manager
Look into the SQL Configuration Manager.
I guess TCP/IP Protocoll is not enabled, so you can Access your SQL Server via. Shared Memory.
So you would be able to Connect if you are on the SQL Server it self but can not Access it from another Maschine.
Otherwise I would say it is an Network Problem as you see the Connection times out. Maybe a Firewall Software is blocking or your Server can not be reached.
The most likely Cause I expririenced, fitting to oyur discrition is disabled TCP/IP Protocol for SQL Server.