Does apache hive have a contact place where I can ask them questions? I've tried everything I could but it still does not work.
I setup hadoop then hive based on the tutorial
http://doctuts.readthedocs.org/en/latest/hive.html
Then I tried to run hiveserver2 and wrote a python script to interact with it but it would hang when I tried to execute a hive command. However, based on the solution here Requests hang when using Hiveserver2 Thrift Java client
Now when I start hiveserver I still get errors.
Using the hive_service library I get the error
Invalid method name: 'execute'
when I call client.execute
and when I try to use pyhs2 I get this output
Traceback (most recent call last):
File "test1.py", line 8, in <module>
database='default') as conn:
File "/home/sakib/anaconda/lib/python2.7/site-packages/pyhs2/__init__.py", line 7, in connect
return Connection(*args, **kwargs)
File "/home/sakib/anaconda/lib/python2.7/site-packages/pyhs2/connections.py", line 46, in __init__
transport.open()
File "/home/sakib/anaconda/lib/python2.7/site-packages/pyhs2/cloudera/thrift_sasl.py", line 74, in open
status, payload = self._recv_sasl_message()
File "/home/sakib/anaconda/lib/python2.7/site-packages/pyhs2/cloudera/thrift_sasl.py", line 92, in _recv_sasl_message
header = self._trans.readAll(5)
File "/home/sakib/anaconda/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
chunk = self.read(sz-have)
File "/home/sakib/anaconda/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 94, in read
raise TTransportException('TSocket read 0 bytes')
thrift.transport.TTransport.TTransportException: None
Here is my sample python scripts to connect with hive
import sys
from hive_service import ThriftHive
from hive_service.ttypes import HiveServerException
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
try:
print "1111"
transport = TSocket.TSocket('localhost', 10000)
transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
print "2222"
client = ThriftHive.Client(protocol)
transport.open()
print "3333"
client.execute("CREATE TABLE r(a STRING, b INT, c DOUBLE)")
print "4444"
transport.close()
except Thrift.TException, tx:
print '%s' % (tx.message)
and
import pyhs2
with pyhs2.connect(host='localhost',
port=10000,
authMechanism="PLAIN",
user='root',
password='',
database='default') as conn:
with conn.cursor() as cur:
#Show databases
print cur.getDatabases()
Related
I work with SQL Server 2019 on server I face issue when I try to read an Excel file from shared path using python 3.10.
SQL Server exists on server 7.7 and files shared i need to access and read exist on same server .
When I execute reading to Excel file on local server, it is working from path D:\ExportExcel\testData.xlsx.
But when try to read the Excel from a shared Path as below
EXECUTE sp_execute_external_script
#language = N'Python',
#script = N'import pandas as pd
df = pd.read_excel(r"\\192.168.7.7\ExportExcel\testData.xlsx", sheet_name = "Sheet1")
print(df)';
I get an error:
Msg 39004, Level 16, State 20, Line 48
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 48
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "", line 5, in
File "D:\ProgramData\MSSQLSERVER\Temp-PY\Appcontainer1\9D383F5D-F77E-444E-9A82-B8839C8801E3\sqlindb_0.py", line 31, in transform
df = pd.read_excel(r"\192.168.7.7\ExportExcel\testData.xlsx", sheet_name = "Sheet1")
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\pandas\util_decorators.py", line 178, in wrapper
return func(*args, **kwargs)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\pandas\util_decorators.py", line 178, in wrapper
return func(*args, **kwargs)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\pandas\io\excel.py", line 307, in read_excel
io = ExcelFile(io, engine=engine)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\pandas\io\excel.py", line 394, in init
Msg 39019, Level 16, State 2, Line 48
An external script error occurred:
self.book = xlrd.open_workbook(self.io)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\xlrd_init.py", line 111, in open_workbook
with open(filename, "rb") as f:
PermissionError: [Errno 13] Permission denied: '\\192.168.7.7\ExportExcel\testData.xlsx'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 605, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 375, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
How to solve issue above please?
What I tried:
I try to open shared path from run; I can open it and create new file and read and write on same path
I tried to use another tool for reading as openrowset
select *
from OPENROWSET('Microsoft.ACE.OLEDB.12.0', 'Excel 12.0 Xml;Database=\\192.168.7.7\ExportExcel\testData.xlsx;HDR=YES','select * FROM [Sheet1$]')
and it read the Excel file successfully.
Folder path and file have all permission like
network service and owner and administrator and authenticated user and all application package and every one and all these have full control over all that .
Please - what could be the issue?
I tried using python pysharm to run script python
for shared path it read success .
I have created a package allowing a user to write data to either a sqlite or Postgres db. I created a module for connecting to the db and a separate module that provides the writing functionality. In the latter module the write is a straightforward pandas internal function call:
indata.to_sql('pay_' + table, con, if_exists='append', index=False)
Writing to an sqlite db (with connection using 'sqlite3') is successful however when writing to a Postgres db I get the following error:
Traceback (most recent call last):
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1778, in execute
ps = cache['ps'][key]
KeyError: ("SELECT name FROM sqlite_master WHERE type='table' AND name=?;", ((705, 0, <function Connection.__init__.<locals>.text_out at 0x7fc3205fb510>),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/io/sql.py", line 1595, in execute
cur.execute(*args)
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 861, in execute
self._c.execute(self, operation, args)
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1837, in execute
self.handle_messages(cursor)
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1976, in handle_messages
raise self.error
pg8000.core.ProgrammingError: {'S': 'ERROR', 'V': 'ERROR', 'C': '42P01', 'M': 'relation "sqlite_master" does not exist', 'P': '18', 'F': 'parse_relation.c', 'L': '1180', 'R': 'parserOpenTable'}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/io/sql.py", line 1610, in execute
raise_with_traceback(ex)
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/compat/__init__.py", line 46, in raise_with_traceback
raise exc.with_traceback(traceback)
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/io/sql.py", line 1595, in execute
cur.execute(*args)
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 861, in execute
self._c.execute(self, operation, args)
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1837, in execute
self.handle_messages(cursor)
File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1976, in handle_messages
raise self.error
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': {'S': 'ERROR', 'V': 'ERROR', 'C': '42P01', 'M': 'relation "sqlite_master" does not exist', 'P': '18', 'F': 'parse_relation.c', 'L': '1180', 'R': 'parserOpenTable'}
I traced the error to the following file:
/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/io/sql.py
What seems to be happening is that the '.to_sql' function is configured to try to write to a db named 'sqlite_master' at this point in the 'sql.py' file:
def has_table(self, name, schema=None):
# TODO(wesm): unused?
# escape = _get_valid_sqlite_name
# esc_name = escape(name)
wld = "?"
query = (
"SELECT name FROM sqlite_master " "WHERE type='table' AND name={wld};"
).format(wld=wld)
return len(self.execute(query, [name]).fetchall()) > 0
Looking more closely at the errors you can see that the connection is correctly made to the db but that pandas is looking for an sqlite db:
I know that the db name was one I used several half a year ago when I first started working with sqlite so I'm thinking that somewhere I set a configuration value. So:
is my reasoning correct?
if so, how do I change the configuration?
if not, what is possibly going on?
Per pandas.DataFrame.to_sql documentation:
con : sqlalchemy.engine.Engine or sqlite3.Connection
Using SQLAlchemy makes it possible to use any DB supported by that
library. Legacy support is provided for sqlite3.Connection objects.
This means only SQLite allows a raw connection for the to_sql method. All other RDBMs including Postgres must use an SQLAlchemy connection for this method to create structures and append data. Do note: read_sql does not require SQLAlchemy since it does not make persistent changes.
Therefore, this raw DB-API connection cannot work:
import psycopg2
con = psycopg2.connect(host="localhost", port=5432, dbname="mydb", user="myuser", password="mypwd")
indata.to_sql('pay_' + table, con, if_exists='append', index=False)
However, this SQLAlchemy connection can work:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://myuser:mypwd#localhost:5432/mydb')
indata.to_sql('pay_' + table, engine, if_exists='append', index=False)
Better use SQLAlchemy for both databases, here for SQLite:
engine = create_engine("sqlite:///path/to/mydb.db")
How does the webrtc pipeline get any information about its peers?
This is what I assume what the on_negotiation_needed callback does?
def start_pipeline(self):
self.pipe = Gst.parse_launch(PIPELINE_DESC)
self.webrtc = self.pipe.get_by_name('sendrecv')
**self.webrtc.connect('on-negotiation-needed', self.on_negotiation_needed)**
self.webrtc.connect('on-ice-candidate', self.send_ice_candidate_message)
self.webrtc.connect('pad-added', self.on_incoming_stream)
self.pipe.set_state(Gst.State.PLAYING)
I see that it has the on_negotiation_needed callback but its unclear where the element variable comes from? I looked here: http://blog.nirbheek.in/2018/02/gstreamer-webrtc.html and here: https://github.com/centricular/gstwebrtc-demos and I am still confused as to how this negotiation works? From what I understand there are 2 (or more) peers and both of them must connect to the signaling server, then one of them has to create an offer.
I await for the message from (I assume) the gstreamer webrtcbin on the signaling server:
print (websocket.remote_address)
#get message from client
message = await asyncio.wait_for(websocket.recv(), 3000)
and I get this error when the pipline starts:
('192.168.11.138', 44120)
Error in connection handler
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 674, in transfer_data
message = yield from self.read_message()
File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 742, in read_message
frame = yield from self.read_data_frame(max_size=self.max_size)
File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 815, in read_data_frame
frame = yield from self.read_frame(max_size)
File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 884, in read_frame
extensions=self.extensions,
File "/usr/local/lib/python3.6/dist-packages/websockets/framing.py", line 99, in read
data = yield from reader(2)
File "/usr/lib/python3.6/asyncio/streams.py", line 672, in readexactly
raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 2 expected bytes
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/websockets/server.py", line 169, in handler
yield from self.ws_handler(self, path)
File "signaling_server.py", line 34, in signaling
message = await asyncio.wait_for(websocket.recv(), 3000)
File "/usr/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
return fut.result()
File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 434, in recv
yield from self.ensure_open()
File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 646, in ensure_open
) from self.transfer_data_exc
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason
I cannot say about Python (unfortunately, cannot make Python bindings for GStreamer work on Windows), however, demo works from C# (I just checked).
First you should connect with your browser to https://webrtc.nirbheek.in/, and get the 'Our id' value.
Your Python Gstreamer should connect to wss://webrtc.nirbheek.in:8443, and use the Id value from the browser.
The browser will get the test image stream from the GStreamer, and the GStreamer application will get the Webcam image from the browser.
HTH, Tom
Here's a screenshot:
Problem Statement: Extract data stored in the .sdf file to python.
System config: Win 10 Pro 64-bit, python 3.5.2-64 bit, adodbapi library, SQL CE 3.5
I am fairly new to programming and I have picked up Python as my first language to learn. Currently, I have hit a wall in process of connecting a SQL CE 3.5 .sdf file.
I have used the adodbapi library. I have searched the web extensively over the past week to find a solution to this problem and to make sure that my connection string is correct. I have tried multiple options/solutions provided on stack overflow and https://www.connectionstrings.com/microsoft-sqlserver-ce-oledb-3-5/.
Code:
import adodbapi
cons_str = "Provider=Microsoft.SQLSERVER.MOBILE.OLEDB.3.5;" \
"Data Source=D:\Work\Programming\Python\SQL_DataTransfer\LF.sdf;"\
"Persist Security Info=False;" \
"SSCE:Max Database Size=4091"
connection = adodbapi.connect(cons_str)
print(connection)
Error Message:
Traceback (most recent call last):
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 93, in make_COM_connecter
c = Dispatch('ADODB.Connection') #connect after CoIninialize v2.1.1 adamvan
NameError: name 'Dispatch' is not defined
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 112, in connect
co.connect(kwargs)
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 269, in connect
self.connector = connection_maker()
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 95, in make_COM_connecter
raise api.InterfaceError ("Windows COM Error: Dispatch('ADODB.Connection') failed.")
adodbapi.apibase.InterfaceError: Windows COM Error: Dispatch('ADODB.Connection') failed.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/Work/Programming/Python/SQL_DataTransfer/SQL_CE_reportDB.py", line 8, in
connection = adodbapi.connect(cons_str)
File "D:\Work\Programs\Python35.virtualenvs\sql_output\lib\site-packages\adodbapi\adodbapi.py", line 116, in connect
raise api.OperationalError(e, message)
adodbapi.apibase.OperationalError: (InterfaceError("Windows COM Error: Dispatch('ADODB.Connection') failed.",), 'Error opening connection to "Provoider=Microsoft.SQLSERVER.MOBILE.OLEDB.3.5;Data Source=D:\Work\Programming\Python\SQL_DataTransfer\LF.sdf;Persist Security Info=False;SSCE:Max Database Size=4091"')
At this point any help is much appreciated.
Thank you,
Sincerely,
JD.
Looks like you have a typo:
Provoider => Provider
adodbapi version = '2.6.0.6' depends on pypiwin32 to be installed in your Python environment.
For adodbapi.py, from line 51:
if api.onIronPython:
from System import Activator, Type, DBNull, DateTime, Array, Byte
from System import Decimal as SystemDecimal
from clr import Reference
def Dispatch(dispatch):
type = Type.GetTypeFromProgID(dispatch)
return Activator.CreateInstance(type)
def getIndexedValue(obj,index):
return obj.Item[index]
else: # try pywin32
try:
import win32com.client
import pythoncom
import pywintypes
onWin32 = True
def Dispatch(dispatch):
return win32com.client.Dispatch(dispatch)
except ImportError:
import warnings
warnings.warn("pywin32 package (or IronPython) required for adodbapi.",ImportWarning)
def getIndexedValue(obj,index):
return obj(index)
In my situation, I traced the fact that the Dispatch function was not defined because an ImportError exception is generated at line 62, (import win32com.client) caught in the except block but for some reason the warning message was not displayed in my console.
Try:
pip install pypiwin32
and the ImportError exception described above should not be raised anymore.
I am currently experimenting with Behavioral Driven Development. I am using behave_django with selenium. I get the following output
Creating test database for alias 'default'...
Feature: Open website and print title # features/first_selenium.feature:1
Scenario: Open website # features/first_selenium.feature:2
Given I open seleniumframework website # features/steps/first_selenium.py:2 0.001s
Traceback (most recent call last):
File "/home/vagrant/newproject3/newproject3/venv/local/lib/python2.7/site-packages/behave/model.py", line 1456, in run
match.run(runner.context)
File "/home/vagrant/newproject3/newproject3/venv/local/lib/python2.7/site-packages/behave/model.py", line 1903, in run
self.func(context, *args, **kwargs)
File "features/steps/first_selenium.py", line 4, in step_impl
context.browser.get("http://www.seleniumframework.com")
File "/home/vagrant/newproject3/newproject3/venv/local/lib/python2.7/site-packages/behave/runner.py", line 214, in __getattr__
raise AttributeError(msg)
AttributeError: 'Context' object has no attribute 'browser'
Then I print the title # None
Failing scenarios:
features/first_selenium.feature:2 Open website
0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
0 steps passed, 1 failed, 1 skipped, 0 undefined
Took 0m0.001s
Destroying test database for alias 'default'...
Here is the code:
first_selenium.feature
Feature: Open website and print title
Scenario: Open website
Given I open seleniumframework website
Then I print the title
first_selenium.py
from behave import *
#given('I open seleniumframework website')
def step_impl(context):
context.browser.get("http://www.seleniumframework.com")
#then('I print the title')
def step_impl(context):
title = context.browser.title
assert "Selenium" in title
manage.py
#!/home/vagrant/newproject3/newproject3/venv/bin/python
import os
import sys
sys.path.append("/home/vagrant/newproject3/newproject3/site/v2/features")
import dotenv
if __name__ == "__main__":
path = os.path.realpath(os.path.dirname(__file__))
dotenv.load_dotenv(os.path.join(path, '.env'))
from configurations.management import execute_from_command_line
#from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
I'm not sure what this error means
I know it is a late answer but maybe somebody is going to profit from it:
you need to declare the context.browser (in a before_all/before_scenario/before_feature hook definition or just test method definition) before you use it, e.g.:
context.browser = webdriver.Chrome()
Please note that the hooks must be defined in a separate environment.py module
In my case the browser wasn't installed. That can be a case too. Also ensure path to geckodriver is exposed if you are working with Firefox.