UPDATE BELOW.....
Have automated csv data dumping into our backend and it looks like there are some malformed items buried in the data. There is a job family title that errantly has a \n in between two words. Which is wrecking our data, so that's the problem.
I want to read in the csv as wholetext, regexp_replace the title with the correction, then load this fixed wholetext into a new dataframe as if I loaded up a correct csv to start with.. Here's the madness of where I'm at right now: Lol.
# Import in the functions I need
# from pyspark.sql.functions import col
# Looks like there is a job family title with an issue. There's a carriage return / line feed between two words messing up the csv
# This needs to be patched before we actually pull the data into the dataframes to begin work
data_requisitions_patch0 = spark.read.text('abfss://container#somethingcool.dfs.core.windows.net/Data/brokencsv.csv', wholetext=True)
data_requisitions_patch0.collect()
data_requisitions_new = data_requisitions_patch0
# print(data_requisitions_patch0)
# data_requisitions_patch0.printSchema()
# data_requisitions_patch0.show()
data_requisitions_patch1 = data_requisitions_patch0 \
.withColumn("value", regexp_replace(col('value'), 'Job - Starting\n', 'Job - Starting'))
data_requisitions_patch1.collect()
print('patch0')
data_requisitions_new.count()
print('patch1')
data_requisitions_patch1.count()
# print('Patch0 dataframe: ' + data_requisitions_patch0.count())
# print('Patch0 dataframe: ' + data_requisitions_patch1.count())
# data_requisitions_test0 = spark(data_requisitions_patch1, header=True)
# data_requisitions_test1 = spark.read.csv('abfss://container#somethingcool.dfs.core.windows.net/Data/brokencsv.csv', header=True)
# data_requisitions_test0.count()
# data_requisitions_test0.printSchema()
# data_requisitions_test1.count()
# data_requisitions_test1.printSchema()
It's obviously a mess right now, I'm trying to troubleshoot is the regexp_replace is working, but not having much luck. Then it occurred to me that I have a single row single column dataframe. Now I'm attempting to try to figure how how to take the dataframe post the 'patch' and turn that back into a normal csv'ed dataframe like everything was ok to begin with.
I left in all my testing nonsense, thought was that you might see where my head is... Unsure if that was helpful or not. Links have been faked, obviously.
First off: Am I going in the right direction? No part of this is really working.. I can't even get the counts to work. test1.count() does return... but test0.count() doesn't? I don't even really care about the counts, that's me just trying to figure out why it's not working.
Secondly: Malformed csv -> wholetext dataframe -> regexp fix the problem -> fixed dataframe with correct headers, rows, like normal.
How off am I?
=======
UPDATE
Made some great progress, I ended up splitting the wholetext dataframe on \n line feeds and exploded that into rows. That works great. Now the dataframe has exactly how many rows it's supposed to have. Now working on trying to figure out how to re-map the columns to get those created correctly.
Thoughts are to take in the header row and try to use that as a map? I don't know, still researching.
I wasn't approaching this right... Was handling this like a typical C# project, pull data from the db and process. But this doesn't really deal well with that. Ended up putting the processed data into the dataframe itself and ran my if checks from contained columns. Works fantastic, and it's a lot faster than trying to extract the data to do the checks.
Related
I have a job which processes files and then lands them as single CSVs on a blob storage container. The problem I face is that I also need to land empty files, which only contain the header. How can this be achieved when I use .saveSingleFile?
Example Code snipped:
df.coalesce(1)
.write
.options(configuration.readAndWriteOptions)
.partitionBy(INGESTION_TIME)
.format("csv")
.mode("append")
.saveSingleFile(path.toString)
Example readAndWriteOptions:
{"sep": ";", "header": "true"}
In other words:
In above case, if df.show() is only displaying a header, no CSV file is written. However, I want to output a csv file without data but column names. Is there an option which would allow this ? Both cases need to be possible, if data is available and if data is not available, therefore something like .take(1) will not be a sufficient solution.
Update:
Looks like this is related to a Spark API Bug and should have been resolved with Version 3.
I want to diagonse autism and one of the best autism diagnosis datasets is ABIDE (Autism Brain Imaging Data Exchange). My final goal in this part is to have a connectivity matrix which I can use for the rest of my research. At this moment I am trying to download ABIDE dataset using Nilearn library. I use the code below as mentioned in Nilearn's documentry, but unfortunately I get an empty list from dataset.func_preproc. I don't know the reason, because it works fine for other datasets of Nilearn which I tested.
dataset = nilearn.datasets.fetch_abide_pcp(derivatives=['func_preproc', 'func_mean'])
print(dataset['func_preproc'])
and the output is:
[]
as you see , dataset['func_preproc'] is an empty list.
Does anyone have any idea about this?
I have a dataframe called updatesDF as follows,
+-------+---------+--------------+-------------------+----------------+-------------------+----------------+
|PartyID| TIN|SourceSystemID| ODSInsertDate| ODSInsertBy| ODSUpdateDate| ODSUpdateBy|
+-------+---------+--------------+-------------------+----------------+-------------------+----------------+
| 11111|222222222| 1|2021-07-20 01:56:25|sneha|2021-07-20 01:56:25|sneha|
+-------+---------+--------------+-------------------+----------------+-------------------+----------------
so updatesDF.show() gives me the above output. Now I need to write this dataframe to a mount path,
updatesDF.write.format('delta').mode('append').save('/mnt/Sneha/Updates/')
So as soon as I write into this location, the updatesDF turns blank like this
+-------+---+--------------+-------------+-----------+-------------+-----------+
|PartyID|TIN|SourceSystemID|ODSInsertDate|ODSInsertBy|ODSUpdateDate|ODSUpdateBy|
+-------+---+--------------+-------------+-----------+-------------+-----------+
+-------+---+--------------+-------------+-----------+-------------+-----------+
There are no other steps in between and I also tried taking backup of this DF...both backup and original DF turns empty after the append step. Please help
It's a weird situation. Looks like a memory saturation.
Can you try updatesDF.persist() or .cache() just after creation.
The following code successfully creates a spreadsheet from a pivot table:
pivot_table_name.to_excel('spreadsheet_file_name.xlsx')
But now I need to be able to create more than one tab. Other posts have recommended using code like this:
for row in dataframe_to_rows('table_name', index=False, header=True):
ws.append(row)
But that is for a "normal" table/dataframe, i.e. not pivoted so if I try that I will of course get the expected:
AttributeError: 'str' object has no attribute '_data'
Is there a way to do this using Pandas?
Also, in case is helpful in any way, I am currently using Papermill to execute multiple scripts so if separate processing is required that is not a problem. Not everything needs to happen in the same script.
If you're trying to add multiple sheets in the same workbook this should work.
with pd.ExcelWriter('spreadsheet_file_name.xlsx') as writer:
pivot_table_name.to_excel(writer, sheet_name='Sheet_name_1')
pivot_table_name2.to_excel(writer, sheet_name='Sheet_name_2')
I'm trying to use graphlab.linear_regression.create and I get an error that I have missing data in the column I am using to predict my model and it says to use dropna to fix the problem. I use dropna but it doesn't get rid of any of the rows missing values. I am typing ti_train.dropna() to try and drop the missing data and age_model = graphlab.linear_regression.create(ti_train, target='Survived', features=['Age'],validation_set=None) for my linear regression. I've also tried fillna with ti_train.fillna('Age', np.median(ti_train['Age'])). I got my data by reading a csv file into an SFrame. Thank you
just executing this code won't modify your sframe (assuming you are running ipython notebook with sframe)
ti_train.fillna('Age', np.median(ti_train['Age']))
Have you tried this (LHS for modifying the sframe) and then try running regression
ti_train = ti_train.fillna('Age', np.median(ti_train['Age']))