Loading NYS ALL Voters (Voter File) into BigQuery crashes - google-bigquery

Thanks for Big Query.
When trying to load the NYS Voter File into Big Query we see this:
The file is
-rw-rw-r-- 1 fedex1 fedex1 5.3G Jun 3 06:43 AllNYSVoters.txt
Is 5.3 Gigs too large to load from the command line?
It is only 15,486,253 rows (15 Million rows)
It appears stackoverflow does not like code only questions, so I am giving more details.
> You have encountered a bug in the BigQuery CLI. Google engineers
> monitor and answer questions on Stack Overflow, with the tag
> google-bigquery:
> http://stackoverflow.com/questions/ask?tags=google-bigquery Please
> include a brief description of the steps that led to this issue, as
> well as the following information:
>
> ========================================
> == Platform == CPython:2.7.3:Linux-3.8.11-x86_64-with-Ubuntu-12.04-precise
> == bq version ==
> 2.0.21
> == Command line == ['/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bq.py',
> '--credential_file',
> '/home/x1/.config/gcloud/legacy_credentials/x7/singlestore.json',
> '--project', 'personal-real-estate', 'load', 'nys.all_voter_file',
> 'AllNYSVoters.txt',
> 'LASTNAME:string,FIRSTNAME:string,MIDDLENAME:string,NAMESUFFIX:string,RADDNUMBER:string,RHALFCODE:string,RAPARTMENT:string,RPREDIRECTION:string,RSTREETNAME:string,RPOSTDIRECTION:string,RCITY:string,RZIP5:string,RZIP4:string,MAILADD1:string,MAILADD2:string,MAILADD3:string,MAILADD4:string,DOB:string,GENDER:string,ENROLLMENT:string,OTHERPARTY:string,COUNTYCODE:integer,ED:integer,LD:integer,TOWNCITY:string,WARD:string,CD:integer,SD:integer,AD:integer,LASTVOTEDDATE:timestamp,PREVYEARVOTED:string,PREVCOUNTY:string,PREVADDRESS:string,PREVNAME:string,COUNTYVRNUMBER:string,REGDATE:timestamp,VRSOURCE:string,IDREQUIRED:string,IDMET:string,STATUS:string,REASONCODE:string,INACT_DATE:timestamp,PURGE_DATE:timestamp,SBOEID:string,VoterHistory:string']
> == UTC timestamp == 2014-06-11 11:42:34
> == Error trace == File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bq.py",
> line 806, in RunSafely
> return_value = self.RunWithArgs(*args, **kwds) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bq.py",
> line 1047, in RunWithArgs
> job = client.Load(table_reference, source, schema=schema, **opts) File
> "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 2045, in Load
> upload_file=upload_file, **kwds) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 1642, in ExecuteJob
> job_id=job_id) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 1627, in RunJobSynchronously
> upload_file=upload_file, job_id=job_id) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 1535, in StartJob
> projectId=project_id).execute() File "/var/host/media/removable/USB_Drive/google-cloud-sdk/platform/bq/bigquery_client.py",
> line 308, in execute
> return super(BigqueryHttp, self).execute(**kwds) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py",
> line 132, in positional_wrapper
> return wrapped(*args, **kwargs) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/apiclient/http.py",
> line 688, in execute
> _, body = self.next_chunk(http=http, num_retries=num_retries) File
> "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py",
> line 132, in positional_wrapper
> return wrapped(*args, **kwargs) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/apiclient/http.py",
> line 867, in next_chunk
> headers=headers) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py",
> line 132, in positional_wrapper
> return wrapped(*args, **kwargs) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/client.py",
> line 490, in new_request
> redirections, connection_type) File "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py",
> line 1586, in request
> (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File
> "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py",
> line 1333, in _request
> (response, content) = self._conn_request(conn, request_uri, method, body, headers) File
> "/var/host/media/removable/USB_Drive/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py",
> line 1289, in _conn_request
> response = conn.getresponse() File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
> response.begin() File "/usr/lib/python2.7/httplib.py", line 407, in begin
> version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
> line = self.fp.readline() File "/usr/lib/python2.7/socket.py", line 430, in readline
> data = recv(1) File "/usr/lib/python2.7/ssl.py", line 241, in recv
> return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 160, in read
> return self._sslobj.read(len)
> ========================================
>
> Unexpected exception in load operation: [Errno 104] Connection reset
> by peer

The last line of the log seems to show what the problem was:
[Errno 104] Connection reset by peer
Was there any interruption in the network connectivity?

Related

Incorrect padding when I want to connect google sheets with streamlit

I am trying to connect streamlit app to the google sheets as guided by the streamlit community. after running the following part of the code:
> scope=[“https://www.googleapis.com/auth/drive”,
“https://www.googleapis.com/auth/spreadsheets”]
credentials = service_account.Credentials.from_service_account_info(
st.secrets[“gcp_service_account”],
scopes=scope,
)
I face the following error.
> 2022-08-22 16:03:54.453 Uncaught app exception Traceback (most recent
> call last): File
> "C:\Users\s.kian\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py",
> line 556, in _run_script
> exec(code, module.__dict__) File "C:\Users\s.kian\OneDrive - Intrum Law\Desktop\tt\gsh.py", line 11, in <module>
> credentials = service_account.Credentials.from_service_account_info( File
> "C:\Users\s.kian\AppData\Local\Programs\Python\Python310\lib\site-packages\google\oauth2\service_account.py",
> line 221, in from_service_account_info
> signer = _service_account_info.from_dict( File "C:\Users\s.kian\AppData\Local\Programs\Python\Python310\lib\site-packages\google\auth\_service_account_info.py",
> line 58, in from_dict
> signer = crypt.RSASigner.from_service_account_info(data) File "C:\Users\s.kian\AppData\Local\Programs\Python\Python310\lib\site-packages\google\auth\crypt\base.py",
> line 113, in from_service_account_info
> return cls.from_string( File "C:\Users\s.kian\AppData\Local\Programs\Python\Python310\lib\site-packages\google\auth\crypt\_python_rsa.py",
> line 154, in from_string
> marker_id, key_bytes = pem.readPemBlocksFromFile( File "C:\Users\s.kian\AppData\Local\Programs\Python\Python310\lib\site-packages\pyasn1_modules\pem.py",
> line 44, in readPemBlocksFromFile
> substrate = ''.encode().join([base64.b64decode(x.encode()) for x in certLines]) File
> "C:\Users\s.kian\AppData\Local\Programs\Python\Python310\lib\site-packages\pyasn1_modules\pem.py",
> line 44, in <listcomp>
> substrate = ''.encode().join([base64.b64decode(x.encode()) for x in certLines]) File
> "C:\Users\s.kian\AppData\Local\Programs\Python\Python310\lib\base64.py",
> line 87, in b64decode
> return binascii.a2b_base64(s) binascii.Error: Incorrect padding
I searched a lot in the internet, not found anything.
try:
> scope=["https://www.googleapis.com/auth/drive",
"https://www.googleapis.com/auth/spreadsheets"]
credentials = service_account.Credentials.from_service_account_info(
st.secrets["gcp_service_account"],
scopes=scope,
)
no idea what this does or where it goes but maybe the error is from “ ” so try replacing it with "

Django Compressor returns error when applied

I have been implementing this active search with htmx. I changed my settings after which my app couldn't work here is part of my settings
COMPRESS_FILTERS = {'css': ['libman.admin.PostCSSFilter']}
COMPRESS_ROOT = BASE_DIR / 'static'
COMPRESS_ENABLED = True
STATICFILES_FINDERS = [
'compressor.finders.CompressorFinder',
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',]
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_DIRS = [os.path.join(BASE_DIR,'static')]
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
I already have compressor added to my installed apps
This is what I have in my libman app admin.
from compressor.filters import CompilerFilter
class PostCSSFilter(CompilerFilter):
command = 'postcss'
When I applied these two to my base file, I got an error.
{% compress css %}
<link rel="stylesheet" href="{% static 'src/main.css' %}">
{% endcompress %}
<!-- new -->
{% compress js %}
<script type="text/javascript" src="{% static 'src/htmx.js' %}"></script>
{% endcompress %}
I got the error below. Would anyone help me debug it??
C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\views\debug.py:420: ExceptionCycleWarning: Cycle in the exception chain detected: exception 'Error 10061 connecting to 127.0.0.1:6379. No connection could be made because the target machine actively refused it.' encountered again.
warnings.warn(
Internal Server Error: /library/home
Traceback (most recent call last):
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django_redis\cache.py", line 31, in _decorator
return method(self, *args, **kwargs)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django_redis\cache.py", line 98, in _get
return self.client.get(key, default=default, version=version, client=client)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django_redis\client\default.py", line 260, in get
raise ConnectionInterrupted(connection=client) from e
django_redis.exceptions.ConnectionInterrupted: Redis ConnectionError: Error 10061 connecting to 127.0.0.1:6379. No connection could be made because the target machine actively refused it.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\handlers\exception.py", line 47, in inner
response = get_response(request)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\handlers\base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\contrib\auth\decorators.py", line 21, in _wrapped_view
return view_func(request, *args, **kwargs)
File "D:\Python\Django\Completed Projects\lib_system\Library-System\libman\views.py", line 273, in index
return render(request, 'libman/home.html',{'books':books,'students':students,
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\shortcuts.py", line 19, in render
content = loader.render_to_string(template_name, context, request, using=using)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\loader.py", line 62, in render_to_string
return template.render(context, request)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\backends\django.py", line 61, in render
return self.template.render(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\base.py", line 170, in render
return self._render(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\base.py", line 162, in _render
return self.nodelist.render(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\base.py", line 938, in render
bit = node.render_annotated(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\base.py", line 905, in render_annotated
return self.render(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\loader_tags.py", line 150, in render
return compiled_parent._render(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\base.py", line 162, in _render
return self.nodelist.render(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\base.py", line 938, in render
bit = node.render_annotated(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\template\base.py", line 905, in render_annotated
return self.render(context)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\compressor\templatetags\compress.py", line 143, in render
return self.render_compressed(context, self.kind, self.mode, forced=forced)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\compressor\templatetags\compress.py", line 111, in render_compressed
cache_key, cache_content = self.render_cached(compressor, kind, mode)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\compressor\templatetags\compress.py", line 89, in render_cached
cache_key = get_templatetag_cachekey(compressor, mode, kind)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\compressor\cache.py", line 104, in get_templatetag_cachekey
"templatetag.%s.%s.%s" % (compressor.cachekey, mode, kind))
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\utils\functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\compressor\base.py", line 200, in cachekey
[self.content] + self.mtimes).encode(self.charset), 12)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django\utils\functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\compressor\base.py", line 193, in mtimes
return [str(get_mtime(value))
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\compressor\base.py", line 193, in <listcomp>
return [str(get_mtime(value))
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\compressor\cache.py", line 110, in get_mtime
mtime = cache.get(key)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django_redis\cache.py", line 91, in get
value = self._get(key, default, version, client)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django_redis\cache.py", line 38, in _decorator
raise e.__cause__
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\django_redis\client\default.py", line 258, in get
value = client.get(key)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\redis\client.py", line 1606, in get
return self.execute_command('GET', name)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\redis\client.py", line 898, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\redis\connection.py", line 1192, in get_connection
connection.connect()
File "C:\Users\Ptar\AppData\Local\Programs\Python\Python39\lib\site-packages\redis\connection.py", line 563, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 10061 connecting to 127.0.0.1:6379. No connection could be made because the target machine actively refused it.

SSL3_GET_RECORD:wrong version number when loading to bigquery table from google cloud storage

When loading to a Bigquery table from files in Google cloud storage, I kept getting this SSL3_GET_RECORD:wrong version number exception.
Eventually when I check the job history from the Google Bigquery job history webpage, the load job will show that it has succeeded.
Could you please help? Thank you.
Here is the error message I am getting:
========================================
== Platform ==
CPython:2.7.6:Linux-2.6.18-194.32.1.el5-x86_64-with-redhat-5.5-Final
== bq version ==
2.0.18
== Command line ==
['/opt/google-cloud-sdk/platform/bq/bq.py', '--credential_file', '/offworld/hornet/.config/gcloud/legacy_credentials/clok#vindicotech.com/singlestore.json', '--project', 'formal-cascade-571', 'load', '--source_format=NEWLINE_DELIMITED_JSON', 'dw_sandbox.impressions_20140603', 'gs://dw_sandbox/impressions/20140603/20140604175042285_20140604195938608_20140603_0_*', '/offworld/specificmedia/logsTobq/schemas/impressionsSchema.txt']
== UTC timestamp ==
2014-06-05 01:19:06
== Error trace ==
File "/opt/google-cloud-sdk/platform/bq/bq.py", line 779, in RunSafely
return_value = self.RunWithArgs(*args, **kwds)
File "/opt/google-cloud-sdk/platform/bq/bq.py", line 1020, in RunWithArgs
job = client.Load(table_reference, source, schema=schema, **opts)
File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 2011, in Load
upload_file=upload_file, **kwds)
File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 1611, in ExecuteJob
job_id=job_id)
File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 1599, in RunJobSynchronously
result = self.WaitJob(job_reference)
File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 1713, in WaitJob
done, job = self.PollJob(job_reference, status=status, wait=wait)
File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 1752, in PollJob
job = self.apiclient.jobs().get(**dict(job_reference)).execute()
File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 307, in execute
return super(BigqueryHttp, self).execute(**kwds)
File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/apiclient/http.py", line 716, in execute
body=self.body, headers=self.headers)
File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/oauth2client/client.py", line 490, in new_request
redirections, connection_type)
File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py", line 1586, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py", line 1333, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/google-cloud-sdk/bin/bootstrapping/../../lib/httplib2/__init__.py", line 1289, in _conn_request
response = conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib64/python2.7/ssl.py", line 241, in recv
return self.read(buflen)
File "/usr/lib64/python2.7/ssl.py", line 160, in read
return self._sslobj.read(len)
========================================
Unexpected exception in load operation: [Errno 1] _ssl.c:1426:
error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
Is there any chance that you're using the same HTTP object in multiple threads? I.e the thread you create the job in is not necessarily the same thread you poll for completion in? If this is the case, this came up internally within google today, and this was the fix:
class _HTTPFactoryWrapper(object):
"""Wraps a request factory so that each request returns a new http object.
API client's Http object is not threadsafe since calls to the same domain will
reuse the same HTTPConnection. If one API call is outstanding then a second
will try to send a request over the same domain. This causes chaos that
seems to surface itself as SSLErrors during processing.
"""
def __init__(self, factory):
self.factory = factory
def request(self, *args, **kwargs):
return self.factory.Create().request(*args, **kwargs)
Then change the creation of the bigquery stub from:
return discovery.build(api_name, api_version, http=http_factory.Create())
To
http_wrapper = _HTTPFactoryWrapper(http_factory)
return discovery.build(api_name, api_version, http=http_wrapper)

Unable to load data into bigquery (BadStatusLine)

I am trying to load a local file into an existing table within bigquery. Tried 3 times on different days. File has 1.1m rows. I can't spot any specific error being encountered. Following are the details spit out...
== Platform ==
CPython:2.7.4:Linux-2.6.18-308.11.1.el5.centos.plus-x86_64-with-redhat-5.8-Final
== bq version ==
v2.0.12
== Command line ==
['/opt/./python2.7.4/bin/bq', 'load', '395733598146:apache_l1.sjc_web_201304', 'x.2013-04-23']
== UTC timestamp ==
2013-05-01 18:48:17
== Error trace ==
File "build/bdist.linux-x86_64/egg/bq.py", line 652, in RunSafely
return_value = self.RunWithArgs(*args, **kwds)
File "build/bdist.linux-x86_64/egg/bq.py", line 880, in RunWithArgs
job = client.Load(table_reference, source, schema=schema, **opts)
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 1634, in Load
upload_file=upload_file, **kwds)
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 1366, in ExecuteJob
job_id=job_id)
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 1352, in RunJobSynchronously
upload_file=upload_file, job_id=job_id)
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 1346, in StartJob
projectId=project_id).execute()
File "build/bdist.linux-x86_64/egg/bigquery_client.py", line 274, in execute
return super(BigqueryHttp, self).execute(**kwds)
File "build/bdist.linux-x86_64/egg/oauth2client/util.py", line 120, in positional_wrapper
return wrapped(*args, **kwargs)
File "build/bdist.linux-x86_64/egg/apiclient/http.py", line 656, in execute
_, body = self.next_chunk(http=http)
File "build/bdist.linux-x86_64/egg/oauth2client/util.py", line 120, in positional_wrapper
return wrapped(*args, **kwargs)
File "build/bdist.linux-x86_64/egg/apiclient/http.py", line 784, in next_chunk
headers=headers)
File "build/bdist.linux-x86_64/egg/oauth2client/util.py", line 120, in positional_wrapper
return wrapped(*args, **kwargs)
File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 428, in new_request
redirections, connection_type)
File "/opt/python2.7.4/lib/python2.7/site-packages/httplib2-0.8-py2.7.egg/httplib2/__init__.py", line 1570, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/opt/python2.7.4/lib/python2.7/site-packages/httplib2-0.8-py2.7.egg/httplib2/__init__.py", line 1317, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/python2.7.4/lib/python2.7/site-packages/httplib2-0.8-py2.7.egg/httplib2/__init__.py", line 1286, in _conn_request
response = conn.getresponse()
File "/opt/python2.7.4/lib/python2.7/httplib.py", line 1045, in getresponse
response.begin()
File "/opt/python2.7.4/lib/python2.7/httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "/opt/python2.7.4/lib/python2.7/httplib.py", line 373, in _read_status
raise BadStatusLine(line)
BigQuery doesn't like uploading large local files directly. Try first uploading it into a google cloud storage bucket (gs://), and then importing it into BQ from there.
Install gsutil directions from command line, or use your Google Developer's console in your web browser
You can able to load a local file into an exiting BigQuery table:
All Rows:
bq load --source_format=CSV mydataset.mytable myfile.csv col1:INTEGER,col2:STRING
Skip First Row:
bq load --skip_leading_rows=1 --source_format=CSV mydataset.mytable myfile.csv col1:INTEGER,col2:STRING

hg push error and username not specified in .hg/hgrc. Keyring will not be used

I did the following:
hg clone ...somelink.to.repo.in.hg... Giga
cd Giga
ls (...it shows me giga.txt file exist in Giga directory)
vi giga.txt (...made some changes..)
hg commit -m "byte"
hg out (got the following error)
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or mercurial#selenic.com
** Mercurial Distributed SCM (version 1.5)
** Extensions loaded: acl, bugzilla, children, churn, color, convert, extdiff, fetch, gpg, graphlog, hgcia, hgk, highlight, interhg, keyword, mercurial_keyring, mq, notify, pager, patchbomb, progress, purge, rebase, record, relink, schemes, share, transplant, zeroconf
Traceback (most recent call last):
File "/usr/bin/hg", line 27, in <module>
mercurial.dispatch.run()
File "/usr/lib/python2.6/site-packages/mercurial/dispatch.py", line 16, in run
sys.exit(dispatch(sys.argv[1:]))
File "/usr/lib/python2.6/site-packages/mercurial/dispatch.py", line 30, in dispatch
return _runcatch(u, args)
File "/usr/lib/python2.6/site-packages/mercurial/dispatch.py", line 47, in _runcatch
return _dispatch(ui, args)
File "/usr/lib/python2.6/site-packages/mercurial/dispatch.py", line 466, in _dispatch
return runcommand(lui, repo, cmd, fullargs, ui, options, d)
File "/usr/lib/python2.6/site-packages/mercurial/dispatch.py", line 336, in runcommand
ret = _runcommand(ui, options, cmd, d)
File "/usr/lib/python2.6/site-packages/mercurial/extensions.py", line 128, in wrap
return wrapper(origfn, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/hgext/pager.py", line 66, in pagecmd
return orig(ui, options, cmd, cmdfunc)
File "/usr/lib/python2.6/site-packages/mercurial/dispatch.py", line 517, in _runcommand
return checkargs()
File "/usr/lib/python2.6/site-packages/mercurial/dispatch.py", line 471, in checkargs
return cmdfunc()
File "/usr/lib/python2.6/site-packages/mercurial/dispatch.py", line 465, in <lambda>
d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
File "/usr/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/extensions.py", line 116, in wrap
util.checksignature(origfn), *args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/hgext/color.py", line 352, in nocolor
return orig(*args, **opts)
File "/usr/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/extensions.py", line 116, in wrap
util.checksignature(origfn), *args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/hgext/mq.py", line 2648, in mqcommand
return orig(ui, repo, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/extensions.py", line 116, in wrap
util.checksignature(origfn), *args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/hgext/graphlog.py", line 365, in graph
return orig(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/util.py", line 401, in check
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/mercurial/commands.py", line 2275, in outgoing
other = hg.repository(cmdutil.remoteui(repo, opts), dest)
File "/usr/lib/python2.6/site-packages/mercurial/hg.py", line 82, in repository
repo = _lookup(path).instance(ui, path, create)
File "/usr/lib/python2.6/site-packages/mercurial/httprepo.py", line 271, in instance
inst.between([(nullid, nullid)])
File "/usr/lib/python2.6/site-packages/mercurial/httprepo.py", line 190, in between
d = self.do_read("between", pairs=n)
File "/usr/lib/python2.6/site-packages/mercurial/httprepo.py", line 134, in do_read
fp = self.do_cmd(cmd, **args)
File "/usr/lib/python2.6/site-packages/mercurial/httprepo.py", line 85, in do_cmd
resp = self.urlopener.open(req)
File "/usr/lib/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.6/urllib2.py", line 429, in error
result = self._call_chain(*args)
File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib/python2.6/urllib2.py", line 855, in http_error_401
url, req, headers)
File "build/bdist.linux-i686/egg/mercurial_keyring.py", line 339, in basic_http_error_auth_reqed
File "/usr/lib/python2.6/urllib2.py", line 833, in http_error_auth_reqed
return self.retry_http_basic_auth(host, req, realm)
File "/usr/lib/python2.6/urllib2.py", line 836, in retry_http_basic_auth
user, pw = self.passwd.find_user_password(realm, host)
File "build/bdist.linux-i686/egg/mercurial_keyring.py", line 333, in find_user_password
File "build/bdist.linux-i686/egg/mercurial_keyring.py", line 184, in find_auth
File "build/bdist.linux-i686/egg/mercurial_keyring.py", line 67, in get_http_password
File "/usr/local/lib/python2.6/site-packages/keyring/core.py", line 37, in get_password
return _keyring_backend.get_password(service_name, username)
File "/usr/local/lib/python2.6/site-packages/keyring/backend.py", line 143, in get_password
items = gnomekeyring.find_network_password_sync(username, service)
gnomekeyring.IOError
My ~/.hgrc (OpenSUSE machine)
[ui]
username=c123456 <Arun.Sangal#MyCompany.com>
[extensions]
mercurial_keyring = /root/mercurial_keyring.py
#[trusted]
#users = *
#groups = *
[extensions]
acl =
bugzilla =
children =
churn =
color =
convert =
eol = !
extdiff =
factotum = !
fetch =
gpg =
graphlog =
hgcia =
hgcr-gui-qt = !
hgk =
highlight =
interhg =
keyword =
largefiles = !
mercurial_keyring =
mq =
notify =
pager =
patchbomb =
perfarce = !
progress =
projrc = !
purge =
rebase =
record =
relink =
schemes =
....
........etc
My local repository
(on OpenSuse cloned folder - inside: /Giga/.hg/hgrc) is:
[paths]
default = http://the.hg.server.com/hg/TestHgRepo1/
myrepo = http://the.hg.server.com/hg/TestHgRepo1/
[auth]
myrepo.schemes = http https
myrepo.prefix = the.hg.server.com/hg
myrepo.username = c123456
I tried everything but this Keyring thing is not working. I get prompt everytime I do:
hg out
hg push
etc hg operation but not when I do
hg commit
Can someone please tell what the heck I'm missing here. Tried the same excercise on Windows with TortoiseHg, with C:...\mercurial.ini (Windows side kinda of unix ~/.hgrc file).. and updated/made sure local repository cloned folder's ../clonedfolder/.hg/hgrc file contains the similar [auth] ..3 lines but Mercurial on Linux OpenSUSE and on Windows using TortoiseHg is not working with keyring.
It's prompting me for entering user credentials again n again :((
can someone pls correct me on what should I do to get this resolved.
if prompted multiple times for user credentials in mercurial. Setup Mercurial_Keyring and then
this question comes which nobody explained in an easy way.
??? how to make the [auth] xx.prefix = servername/hg_or_something work for all repositories under servername/hg location either if I use servername, servername's IP or servername's FQDN ?
Final ANSWER: Arun • 2 minutes ago −
OK, I put this in ~/.hgrc (Linux/Unix -home directory's .hgrc hidden file) or Windows users %UserProfile%/mercurial.ini or %HOME%/mercurial.ini file.
[auth]
default1.schemes = http https
default1.prefix = hg_merc_server/hg
default1.username = c123456
default2.schemes = http https
default2.prefix = hg_merc_server.company.com/hg
default2.username = c123456
default3.schemes = http https
default3.prefix = 10.211.222.321/hg
default3.username = c123456
Now, I can checkout using either Server/IP/Server's FQDN.