Had ran a overnight job using a script.
It worked for lot of tables and then some 4 hrs back 7am IST approx started behaving weird
Now even single commands give the same error
bq load --max_bad_records=10 tbl163.a_V3_14Jun2012 a_V3_14Jun2012.log.gz ../schema/analyze.schema
Error:
BigQuery error in load operation: Could not connect with BigQuery server, http
response status: 502
Update: I have received the following error just now
You have encountered a bug in the BigQuery CLI. Please send an email to bigquery- team#google.com to report this, with the following information:
========================================
== Platform ==
CPython:2.7.3:Linux-3.2.0-25-virtual-x86_64-with-Ubuntu-12.04-precise
== bq version ==
v2.0.6
== Command line ==
['/usr/local/bin/bq', 'load', '--max_bad_records=10', 'vizvrm299.analyze_VIZVRM299_26Jun2012', 'analyze_VIZVRM299_26Jun2012.log.gz', '../schema/analyze.schema']
== Error trace ==
File "build/bdist.linux-x86_64/egg/bq.py", line 614, in RunSafely
self.RunWithArgs(*args, **kwds)
File "build/bdist.linux-x86_64/egg/bq.py", line 791, in RunWithArgs
job = client.Load(table_reference, source, schema=schema, **opts)
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 1473, in Load
upload_file=upload_file, **kwds)
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 1228, in ExecuteJob
job_id=job_id)
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 1214, in RunJobSynchronously
upload_file=upload_file, job_id=job_id)
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 1208, in StartJob
projectId=project_id).execute()
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 184, in execute
return super(BigqueryHttp, self).execute(**kwds)
File "build/bdist.linux-x86_64/egg/apiclient/http.py", line 644, in execute
_, body = self.next_chunk(http)
File "build/bdist.linux-x86_64/egg/apiclient/http.py", line 708, in next_chunk
raise ResumableUploadError("Failed to retrieve starting URI.")
========================================
Unexpected exception in load operation: Failed to retrieve starting URI.
Just to close out this question, are you still observing these errors? This looks like a network configuration issue, not directly an issue with the bq command line tool. However, AFAIK, bq doesn't provide functions for resuming job insertion if there is a problem with the network.
Related
I work with SQL Server 2019 on server I face issue when I try to read an Excel file from shared path using python 3.10.
SQL Server exists on server 7.7 and files shared i need to access and read exist on same server .
When I execute reading to Excel file on local server, it is working from path D:\ExportExcel\testData.xlsx.
But when try to read the Excel from a shared Path as below
EXECUTE sp_execute_external_script
#language = N'Python',
#script = N'import pandas as pd
df = pd.read_excel(r"\\192.168.7.7\ExportExcel\testData.xlsx", sheet_name = "Sheet1")
print(df)';
I get an error:
Msg 39004, Level 16, State 20, Line 48
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 48
An external script error occurred:
Error in execution. Check the output for more information.
Traceback (most recent call last):
File "", line 5, in
File "D:\ProgramData\MSSQLSERVER\Temp-PY\Appcontainer1\9D383F5D-F77E-444E-9A82-B8839C8801E3\sqlindb_0.py", line 31, in transform
df = pd.read_excel(r"\192.168.7.7\ExportExcel\testData.xlsx", sheet_name = "Sheet1")
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\pandas\util_decorators.py", line 178, in wrapper
return func(*args, **kwargs)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\pandas\util_decorators.py", line 178, in wrapper
return func(*args, **kwargs)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\pandas\io\excel.py", line 307, in read_excel
io = ExcelFile(io, engine=engine)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\pandas\io\excel.py", line 394, in init
Msg 39019, Level 16, State 2, Line 48
An external script error occurred:
self.book = xlrd.open_workbook(self.io)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\xlrd_init.py", line 111, in open_workbook
with open(filename, "rb") as f:
PermissionError: [Errno 13] Permission denied: '\\192.168.7.7\ExportExcel\testData.xlsx'
SqlSatelliteCall error: Error in execution. Check the output for more information.
STDOUT message(s) from external script:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 605, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "D:\SQL Data\MSSQL15.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 375, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.
How to solve issue above please?
What I tried:
I try to open shared path from run; I can open it and create new file and read and write on same path
I tried to use another tool for reading as openrowset
select *
from OPENROWSET('Microsoft.ACE.OLEDB.12.0', 'Excel 12.0 Xml;Database=\\192.168.7.7\ExportExcel\testData.xlsx;HDR=YES','select * FROM [Sheet1$]')
and it read the Excel file successfully.
Folder path and file have all permission like
network service and owner and administrator and authenticated user and all application package and every one and all these have full control over all that .
Please - what could be the issue?
I tried using python pysharm to run script python
for shared path it read success .
My streaming job is now failing with the below error, streaming job worked fine for almost 2 months, and it is completely stateless transformation and just needs to append the new rows to the destination delta table. Before streaming, I'm manually providing the schema to a csv files, even verified the streaming job schema and downstream table schema both matches perfectly along with the datatype.
Not sure, why even in the stateless transformation, I'm getting the below error. Any help would be appreciated.
File "/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 2442, in _call_proxy
return_value = getattr(self.pool[obj_id], method)(*params)
File "/databricks/spark/python/pyspark/sql/utils.py", line 195, in call
raise e
File "/databricks/spark/python/pyspark/sql/utils.py", line 192, in call
self.func(DataFrame(jdf, self.sql_ctx), batch_id)
File "<command-422857213447422>", line 2, in write_to_managed_table
print(f"inside foreachBatch for batch_id:{batchId}, rows in passed dataframe: {micro_batch_df.count()}")
File "/databricks/spark/python/pyspark/sql/dataframe.py", line 670, in count
return int(self._jdf.count())
File "/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__
return_value = get_return_value(
File "/databricks/spark/python/pyspark/sql/utils.py", line 110, in deco
return f(*a, **kw)
File "/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o433.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 28 in stage 13792.0
failed 4 times, most recent failure: Lost task 28.3 in stage 13792.0 (TID 752198)
(10.139.64.13 executor 45):
org.apache.spark.sql.execution.streaming.state.StateSchemaNotCompatible: Provided schema
doesn't match to the schema for existing state! Please note that Spark allow difference of
field name: check count of fields and data type of each field.
There might a problem with the CSV file, it could be corrupted.
You can ignore this csv file by setting the "mode" option to "PERMISSIVE" or "DROPMALFORMED".
mode (default PERMISSIVE): allows a mode for dealing with corrupt records during parsing.
PERMISSIVE : sets other fields to null when it meets a corrupted record. When a schema is set by user, it sets null for extra fields.
DROPMALFORMED : ignores the whole corrupted records.
FAILFAST : throws an exception when it meets corrupted records.
https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/streaming/DataStreamReader.html#csv(path:String):org.apache.spark.sql.DataFrame
spark.read.format("csv")
.option("header,"true")
.option("path","your.csv")
.option("mode","DROPMALFORMED")
.schema(csvSchema)
.load()
I have problem debugging the error I got from dask distributed.
I am using loc to take out small dataframe from large ~100M rows and converting to daskArray.
only when I want to grab the data from the cluster I ran into this issue.
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
errors=errors,
*
Finally when my data is ready to be taken out with
ddf2local=client.gather(chunkDelayedDB.result())
*
The worker contains data that will die and I get following message. In one machine with 32 gb it works fine
File "/home/yousef/pipy/lib/python3.6/site-packages/distributed/client.py", line 244, in _result
result = await self.client._gather([self])
File "/home/yousef/pipy/lib/python3.6/site-packages/distributed/client.py", line 1761, in _gather
response = await future
File "/home/yousef/pipy/lib/python3.6/site-packages/distributed/client.py", line 1813, in _gather_remote
response = await self.scheduler.gather(keys=keys)
File "/home/yousef/pipy/lib/python3.6/site-packages/distributed/core.py", line 748, in send_recv_from_rpc
result = await send_recv(comm=comm, op=key, **kwargs)
File "/home/yousef/pipy/lib/python3.6/site-packages/distributed/core.py", line 547, in send_recv
raise exc.with_traceback(tb)
File "/home/admin/mypy36/lib/python3.6/site-packages/distributed/core.py", line 403, in handle_comm
File "/home/admin/mypy36/lib/python3.6/site-packages/distributed/scheduler.py", line 2561, in gather
KeyError: 'delayedDB-42a099cc4ddd139877f60c67295ab09a'
I trackdown the memory of the worker being killed at is fine untill getting the data (also it had alot of data.)
I would acknowledge any support that might help so greatly.
When I run a query for a table around 20Million records the job gets successful in other environments but when I run the same on Windows command line I get an error saying - You have encountered a bug in the BigQuery CLI.
Query : bq --quiet --format=csv --project_id="proj_id" query
--max_rows=999999999 " select ids from dataset.table group each by id"
Do the execution change according to OS? The same query works for 5 Million records
Error log
== UTC timestamp ==
2015-02-19 08:33:16
== Error trace ==
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform/bq\bq.py", line 719, in RunSafely
return_value = self.RunWithArgs(*args, **kwds)
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform/bq\bq.py", line 1086, in RunWithArgs
max_rows=self.max_rows)
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\bq\bigquery_client.py", line 846, in ReadSchemaAndJobRows
return reader.ReadSchemaAndRows(start_row, max_rows)
File "C:\Program Files\Google\Cloud SDK\google-cloud-sdk\platform\bq\bigquery_client.py", line 2200, in ReadSchemaAndRows
rows.append(self._ConvertFromFV(schema, row))
========================================
Migration from plone 3.3.2 to plone 4.2.1 fails with PosKeyError. I've tried recipes from this article http://plonechix.blogspot.com/2009/12/definitive-guide-to-poskeyerror.html.
I've run error_finder snippet, but it didn't give me any execeptions. I've also tried to take object in debugger using app.mysite._p_jar[p64(oid)] - also no success, it fails with the same error.
How can I delete the broken object or at least get more info about object (e.g. its class name or location)?
Full traceback:
POSKeyError('\x00\x00\x00\x00\x00\x0ey=',)
(Also, the following error occurred while attempting to render the standard error message, please see the event log for full details:
An operation previously failed, with traceback:
File "/Users/makmak/Plone/buildout-cache/eggs/Zope2-2.13.16-py2.7.egg/ZServer/PubCore/ZServerPublisher.py", line 31, in __init__
response=b)
File "/Users/makmak/Plone/buildout-cache/eggs/Zope2-2.13.16-py2.7.egg/ZPublisher/Publish.py", line 443, in publish_module
environ, debug, request, response)
File "/Users/makmak/Plone/buildout-cache/eggs/Zope2-2.13.16-py2.7.egg/ZPublisher/Publish.py", line 237, in publish_module_standard
response = publish(request, module_name, after_list, debug=debug)
File "/Users/makmak/Plone/buildout-cache/eggs/Zope2-2.13.16-py2.7.egg/ZPublisher/Publish.py", line 134, in publish
transactions_manager.commit()
File "/Users/makmak/Plone/buildout-cache/eggs/Zope2-2.13.16-py2.7.egg/Zope2/App/startup.py", line 301, in commit
transaction.commit()
File "/Users/makmak/Plone/buildout-cache/eggs/transaction-1.1.1-py2.7.egg/transaction/_manager.py", line 89, in commit
return self.get().commit()
File "/Users/makmak/Plone/buildout-cache/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 336, in commit
t, v, tb = self._saveAndGetCommitishError()
File "/Users/makmak/Plone/buildout-cache/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 329, in commit
self._commitResources()
File "/Users/makmak/Plone/buildout-cache/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 443, in _commitResources
rm.commit(self)
File "/Users/makmak/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-macosx-10.4-x86_64.egg/ZODB/Connection.py", line 572, in commit
oid, serial, transaction)
File "/Users/makmak/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-macosx-10.4-x86_64.egg/ZODB/BaseStorage.py", line 416, in checkCurrentSerialInTransaction
committed_tid = self.getTid(oid)
File "/Users/makmak/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-macosx-10.4-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 770, in getTid
with self._lock:
File "/Users/makmak/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-macosx-10.4-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 403, in _lookup_pos
raise POSKeyError(oid)
POSKeyError: 0x0e793d
You can use the fsrefs.py to find the bad object.
A very short article on using it is: http://nathanvangheem.com/news/fixing-broken-zodb-object-references
I believe this is the same issue I just ran into which happens if a savepoint is rolled back that included adding an object to the catalog. I think this is a bug in the ZODB but you can workaround it by addressing whatever is rolling back a savepoint, in this case that's the migration of files and images to blobs. So if you fix what's keeping those files or images from successfully migrating to BLOBs (or just delete them) then it should succeed.