I'm publishing my CSV files to CKAN using the API. I want to make my data easy to open in Brazilian Excel, so it must have:
semicolon ";" separated columns
coma "," as a decimal separator
use encoding cp-1252
I'm using Data Store and Data Pusher.
My problem is that if I upload my data with encoding cp1252, Data Pusher sends it as is to the Data Store that expects the data as UTF-8. The data preview doesn't display the accents correctly. In the image below Março were the correct value to display:
I want to have my user downloading the data as cp-1252, so it opens easily in Excel, but also have CKAN displaying it correctly. I must specify the encoding of the file while uploading the file.
I couldn't specify the encoding directly, but taking a look at Data Pusher source I saw that it uses the Messy Tables library. MT obey the environment locale set of the host machine, so I configure it to pt_BR.UTF8 and my accents worked fine.
Now I can publish my data with commas as a decimal separator and using encoding Windows-1252. The data correctly opens in Excel when it is downloaded, and is also correctly displayed in the Data Explorer.
Related
Here is a scenario.
We are collecting programmatically some data into Javascript object which contain Polish characters.
Then we are converting it to string with JSON2CSV library and sending to Azure blob with uploadBlockBlob method from #azure/storage-blob library.
After a while we are using Azure Functions triggered by blob storage trigger. We get "Myblob" property with string with CSV content. Then we are using Papaparse library to convert it back to object. And finally we are using content of the object to update database through mssql library.
In the process we are loosing polish characters.
JSON2CSV when converting to string does not seem to have "encoding" property exposed. Nor the uploadBlockBlob. With Papaparse enforcing encoding with "UTF-8" does not have any effect on the process (changing to cp1250 also does not help). The original content is scraped from a web page with software running on WIndows machine.
Any ideas how to preserve the encoding throughout the pipeline?
Closing the question as it appeared to be my mistake. I pushed Varchar type into NVarchar column - that is why non-latin characters got wrong.
I'm having trouble saving BigQuery table data having special characters as json with the proper encoding. This issue is only via the UI. When I save from the CLI the character encoding is proper. In the table the data is correctly encoded. I'm just clicking 'Save Result' and save as JSON local file. While loading data to the table the data was UTF-8 and properly encoded too. Sample special char: Bié
How to save table data with special characters from the UI correctly? I can use cli but its easier to use the UI if I can get this to export correctly.
As discussed in the comments, I could not reproduce the same problem. Since the results returned by the BigQuery Console and CLI should be the same I suggested that this could be a problem in the text editor which was confirmed.
When I try to convert the byte arrays for an Image (& an audio file, created in a Xamarin Forms app) to JSON & save it to an azure App Service SQL backend in a NVARCHAR(Max) column, it completely omits that record & saves a blank string instead.
Here is a sample of the JSON file that gets created.
Large Json Data
when the json file does not have byte arrays, the resulting string data saves fine.
Rest of the data in corresponding columns also saves just fine.
Don't upload and place in the database. Place it in a storage account. For a blog on how to do this (roughly), see https://shellmonger.com/2016/05/30/30-days-of-zumo-v2-azure-mobile-apps-day-27-file-handling-v1/ and https://shellmonger.com/2016/06/01/30-days-of-zumo-v2-azure-mobile-apps-day-28-file-handling-v2/
If you aren't using Xamarin and/or Cordova, there are similar SDKs for v1 handling in iOS and Android. File Sync only works in .NET right now though.
I have a database encoded in utf8 and I want to export it to a dump file. The problem is that when I do it the data in the dump file are not encoded in utf8. Is there a way to define the encoding when creating the dump file ?
Your DB when you created it, may have been using another form of encoding aside from UTF8. You may want to refer to this article about how to change encoding settings. Hopefully once that has been changed you will be able to export.
https://dev.mysql.com/doc/refman/5.0/en/charset-applications.html
This doc will show you how to encode per table, as well as how to change your encoding via CLI.
Cheers.
I want to know what are the possible errors that can arose from Big Query server side during upload mechanism, though the .CSV file that i'm uploading contains perfect data. Can you list out those errors?
Thanks.
Some of the common errors are:
Files must be encoded in UTF-8 format.
Source data must be properly
escaped within standard guidelines for CSV and JSON.
The structure of
records and the data within of must match the schema provided.
Individual files must be under the size limits listed on our
quota/limits page.
More information about BigQuery source data formats.
Check out our Data Loading cookbook for additional tips.