I have exported some data from Paddle in a CSV format. Unfortunately, the CSV format hasn't been recognized because it contains both the date and time. I didn't manage to change the column in Numbers, neither in Google Drive, as the file is way too big. Is there maybe a way how to change the format directly in GoodData? Thanks
No, this is not possible. The data have to be already transformed when loading to the GD. Generally, the way it's formatted in the CSV doesn't seem to be recognized correctly because it contains both date and time.
Here are the requirements for loading the CSVs. There are some online tools, which allow transforming the data - they can even use google sheets. Alternatively, as a workaround, you could set the time as an attribute - in case this column does not need to be treated as a date, but has to be in the platform."
Related
I an using Google Big Query and I have a field, named 'AsOfDate' which is set as a string datatype. I have a bunch of data in this field, which I really want to set as DateTime or just Date. Either is fine. I Googled for a solution, and I thought this would be pretty easy to do, but I can't seem to get the data type updated. I don't want to run a simple select statement; I want to permanently change the Schema. Has anyone run into this and figured out how to do this kind of thing? If so, please share your insights. Thanks!
To quote directly from the official documentation: 'Changing a column's data type is not supported by the BigQuery web UI, the command-line tool, or the API.'
https://cloud.google.com/bigquery/docs/manually-changing-schemas#changing_a_columns_data_type
There are two ways to manually change a column's data type:
Using a SQL query — Choose this option if you are more concerned about
simplicity and ease of use, and you are less concerned about costs.
Recreating the table — Choose this option if you are more concerned
about costs, and you are less concerned about simplicity and ease of
use.
You could use either of the approaches above along with the PARSE_DATE() function to transform your string into a date field.
https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#parse_date
I have a workbook I use to collect relevant data from a different workbook I only have read-rights to. I connected the latter to my own with Excel's built-in function "Data-connection". Copying a worksheet like this and then copying relevant data from the copied sheet is in my case (yes, I tested it) faster (by about 30%) as opposed to opening said workbook and then copying the relevant data directly from source.
The default date format in the source is "dd/mm/yy" (though it obviously adapts to the local date format settings when opening). The copied sheet keeps that date format but when copying the dates with regional settings as "english-US" Excel assumes the source format to be "yy/mm/dd" even though the cells are formatted as "dd/mm/yy".
And here lies the problem:
Not all users of my workbook can use the local date format of "dd/mm/yy", some have to use "mm/dd/yy" or "yy/mm/dd".
How can I tell Excel to copy from a specific date format to the local date format when the local format is not always different?
Checking for the local format with Application.International(xlDateOrder)and then changing the dates if the format differs seems like a complicated workaround which I would like to avoid.
The source format is dd/mm/yy after copying with TargetSheet.Cells(TargetRow, TargetColumn).Value = CDate(SourceSheet.Cells(SourceRow, SourceColumn).Value) the format was dd/mm/yy with days as year and year as days.
It might depend on how exactly you are copying the data.
In your case it isn't entirely clear to me what your source data looks like, but for instances where typecasting dates with CDate isn't sufficient to make Excel infer the correct format by itself, you should be able to force the correct interpretation using Format:
TargetSheet.Range("A1").Value = _
CDate(Format(SourceSheet.Range("A1").Value, "dd/mm/yy"))
Do note that you might encounter datasets where you'll need to tweak this method (for example data with varying formats or particular formats that are ill suited for Excel's algorithm for understanding the conversion).
In some instances (text conversion maybe), you may have to apply the format afterwards. That is, applying Format to a CDate object instead of the other way around. I haven't looked this up in any significant extent and can't provide specific details or examples, but I am mentioning it so that you'll be on the look-out.
I am also unsure to what extent this approach will work for source formats that are outside the "english-US" format that is standard in VBA.
self-taught SAS user here.
I often work with datasets that I have little control over and are shared among several different users.
I generally have been reading in files as CSVs using an infile statement + defining the variables with blocks of informat, format, and input statements. During this process, can I go ahead and rename variables--provided that everything is renamed in the correct order--or do they have to match the original dataset and be renamed in a later data step?
For example, the variable name in the dataset is '100% Fully Paid Out.' I know SAS variables can't start with numbers and I'd also like to simplify variable names in general, so could I do something like the following:
infile statement...
informat Paid $3.;
format Paid $3.;
input Paid $;
run;
Or maybe I'm going about this very inefficiently. I've tried doing simple proc imports without this whole informat/format/input business, but I've found that trying to redefine variable types afterwards causes more of a headache for me (all datasets I work with have combinations of text, dollars, percentages, general numbers, dates...). In any case, other tips highly appreciated--thanks!
EDIT
Maybe the question I should ask is this: is there any way of keeping the format of the csv for dollars and percentages (through proc import, which seems to convert these to characters)? I know I can manually change the formats from dollars/percentages to "general" in Excel prior to importing the file, but I'd prefer avoiding additional manual steps and also because I actually do want to keep these as dollars and percentages. Or am I just better off doing the informat/format/input to specify data types for the csv, so that variables are read in exactly how I want them to be read in?
Note: I've been unable to proc import xls or xlsx files, either because I'm on a 64-bit computer and/or I'm missing required drivers (or both). I was never able to do this even on a 32-bit computer either.
CSV files do not contain any metadata about the variable types, as your note about trying to import them into Excel demonstrates. You can use PROC IMPORT to have SAS make an educated guess as to how to read them, but the answer could vary from file to file based on the particular data values that happen to appear.
If you have data in XLS or XLSX files you should be able to read them directly into SAS using a libname with the XLS or XLSX engine. That does not use Excel and so does not have any conflicts between 32 and 64 installation. In fact you don't even need Excel installed. SAS will do a better job of determining the variable types from Excel files than from CSV files, but since Excel is a free-form spreadsheet you still might not have consistent variable types for the same variable across multiple files. With an Excel spreadsheet you might not even have the same data type consistently in a single column of a single sheet.
You are better off writing your own data step to read the file. That way you can enforce consistency.
What I typically do when given a CSV file is copy the names from the first row and use it to create a LENGTH statement. This will both define the variables and set the order of the variables. You could at this point give the variables new names.
length paid $3 date amount 8 ;
Then for variables that require an INFORMAT to be read properly I add an INFORMAT statement. Normally this is only needed for date/time variables, but it might also be needed if numeric values include commas or percent signs. The DOLLAR. informat is useful if your CSV file has numbers formatted with $ and/or thousands separators.
informat date mmddyy. amount dollar. ;
Then for variables that require a FORMAT to be displayed properly I add a FORMAT statement. Normally this is only needed for date/time variables. It is only required for character variables if you want to attach $CHAR. format in order to preserve leading spaces.
format date yymmdd10. ;
Then the INPUT statement is really easy since you can use a positional variable list. Note that there is no need to include informats or $ in the INPUT statement since the types are already defined by the LENGTH statement.
input paid -- amount ;
Is there any way to QUICKLY convert numbers/dates stored as text (without knowing exactly which cells are affected) to their correct type using VBA.
I get data in an ugly text-deliminated format, and I wrote a macro that basically does text-to-columns on it, but is more robust (regular text-to-columns will not work on my data, and I also don't want to waste time going through the wizard every time...). But, since I have to use arrays to process the data efficiently, everything gets stored as a String (and is thus transferred to the worksheet as text).
I don't want to have to cycle through every cell, as this takes a LONG time (these are huge data files - I need to use arrays to process them). Is there a simple command I can apply to the entire range to do this?
Thanks!
This has to do with the data type of the columns modify the column from general to the correct data type and the placement of text data should get automatically converted... here's an example where I pasted the text 012345 into different columns having different data types. Note how the displayed value is different for the different types but the value is retained (except on number and general which truncate a leading 0.
However if you don't know what field is of what type... you're really out of luck.
There is a way is there. Just multiply 1 with the data in the column have text to converted as number, whether it is text or not it will convert to numbers only.
Read the following the link for more.
http://chandoo.org/wp/2014/09/02/convert-numbers-stored-as-text-tip/
I'm setting up a database using PHPMyAdmin and many fields will be large chunks of HTML.
What MySQL datatype & attributes should be used for fields that store large amounts of HTML data?
TEXT, MEDIUMTEXT, or LONGTEXT
I would recommend against storing large chunks of HTML (or other text data) in a database. I find that it's often far more useful to store files as files, and put a filename in the database instead.
On the other hand, if you're doing everything through phpMyAdmin, you may not have that option available to you.
You really really should start with the documentation, then if you have questions based on the data types you find there, try to ask for some clarification. But it really helps to understand what the datatypes are before asking the question: Documentation here:
http://dev.mysql.com/doc/refman/5.4/en/data-types.html
That said, take a closer look at text and blob. Text will store a large body of textual information (probably a good choice) where blob is designed for binary data. This does make a difference based on the query functions and what data types they operate on.
I think you can store HTML in simple TEXT field. If your html is more then 64KB then you can use MEDIUMTEXT instead.
See also Storage Requirements for String Types for more details about maximum length of stored value.
Also remember than characters in Unicode can require more then 1 byte to store.