How to convert multiple LCI ecospold files to a custom excel format/ how to use parse_file from pyecospold/ how to read ecospold into brightway - lxml

I have multiple ecospold (version 1) files with LCI data that I want to convert to a custom excel format. I need all data given in the ecospold file. For my own convinience I want to use python to complete this task.
My research until now has lead me to the following conclusions:
There exist at least two converters (by GLAD and openLCA) to convert ecospold formats (1 and 2) to e.g. the ILCD. But those formats are not helping me to go anywhere, since I need to have all the data accessible in python and in order to then write it into my custom excel format.
To get the data in python, the package pyecospold (https://github.com/sami-m-g/pyecospold) seems to be a suitable choice.
According to the README that can be found at the pyecospold github repository,
ecoSpold = parse_file("data/v1/v1_1.xml") # Replace with your own XML file
should do the job. So I implemented the following lines:
import os
from pyecospold import parse_file, save_file, Defaults
from lxml import etree
cd = os.getcwd()
path_input = cd + r'\inputs\ecospold_test.xml'
# Parse the required XML file to EcoSpold class.
es = parse_file('inputs/ecospold_test.xml')
Now I run into the error:
TypeError: parse_file() missing 2 required positional arguments: 'schema_path' and 'ecospold_lookup'
I understood that a schema in xsd format is needed, therefore I got the schema files from the github and amended my last line of code:
es = parse_file('inputs/ecospold_test.xml', 'inputs/schemas/v1/EcoSpold01Dataset.xsd')
Now there is still one argument missing:
TypeError: parse_file() missing 1 required positional argument: 'ecospold_lookup'
Since I have no experience in parsing xml files in python, I have no idea what to do with this. Additionally, I am confused why the README does not say anything about those additionally needed arguments.
My second idea was to use brightway to get the data into python. But since brightway itself is quite an extensive package, I could not find a simple (or any) way to do this. (Sadly, the notebooks linked in the answer of this question Import Ecoinvent 2.2 Ecospold files into Brightway do not exist anymore)
Another option would of course be to write my own parser. But because I am lacking experience and pyecospold does exactly this (at least in my understanding), I would like to avoid this option.
Additionally, there in openLCA it is possible to read in ecospold files and then export them to an excel format. From this excel format I could of course make my custom excel format. The problem here is that I have no idea how to automize this, because I do not want to read in and export each file individually and manually in openLCA.
If anyone has an idea on how to solve one of my subproblems or a good alternative on how to solve my general problem, I would be very thankful. :)

Related

Reading *.cdpg file with python without knowing structure

I am trying to use python to read a .cdpg file. It was generated by the labview code. I do not have access to any information about the structure of the file. Using another post I have had some success, but the numbers are not making any sense. I do not know if my code is wrong or if my interpretation of the data is wrong.
The code I am using is:
import struct
with open(file, mode='rb') as file: # b is important -> binary
fileContent = file.read()
ints = struct.unpack("i" * ((len(fileContent) -24) // 4), fileContent[20:-4])
print(ints)
The file is located here. Any guidance would be greatly appreciated.
Thank you,
T
According to the documentation here https://www.ni.com/pl-pl/support/documentation/supplemental/12/logging-data-with-national-instruments-citadel.html
The .cdpg files contain trace data. Citadel stores data in a
compressed format; therefore, you cannot read and extract data from
these files directly. You must use the Citadel API in the DSC Module
or the Historical Data Viewer to access trace data. Refer to the
Citadel Operations section for more information about retrieving data
from a Citadel database.
.cdpg is a closed format containing compressed data. You won't be able to interpret them properly not knowing the file format structure. You can read the raw binary content and this is what you're actually doing with your example Python code

Can I use a .txt file as a user database in Telegram? I use Telethon

So, I created a minigame bot on telegram. The bot just contains a fishing game, and it's already running. I want if a user fishes and gets a fish, the fish will be stored in a database. So the user can see what he got while fishing. Does this is require SQL?
I haven't tried anything, because I don't understand about storing data in python. If there is a tutorial related to this, please share it in the comments. Thank you
You can use anything to store user data, including text files.
The simplest approaches to storing data can be serializing a dictionary to JSON with the builtin json module:
DATABASE = 'database.json' # (name or extension don't actually matter)
import json
# loading
with open(DATABASE, 'r', encoding='utf-8') as fd:
user_data = json.load(fd)
user_data[1234] = 5 # pretend user 1234 scored 5 points
# saving
with open(DATABASE, 'w', encoding='utf-8') as fd:
json.dump(user_data, fd)
This would only support simple data-types. If you need to store custom classes, as long as you don't upgrade your Python version, you can use the built-in pickle module:
DATABASE = 'database.pickle' # (name or extension don't actually matter)
import pickle
# loading
with open(DATABASE, 'rb') as fd:
user_data = pickle.load(fd)
user_data[1234] = 5 # pretend user 1234 scored 5 points
# saving
with open(DATABASE, 'wb') as fd:
pickle.dump(user_data, fd)
Whether this is a good idea or not depends on how many users you expect your bot to have. If it's even a hundred, these approaches will work just fine. If it's in the thousands, perhaps you could use separate files per user, and still be okay. If it's more than that, then yes, using any database, including the built-in sqlite3 module, would be a better idea. There are many modules for different database engines, but using SQLite is often enough (and there are also libraries that make using SQLite easier).
Telethon itself uses the sqlite3 module to store the authorization key, and a cache for users it has seen. It's not recommended to reuse that same file for your own needs though. It's better to create your own database file if you choose to use sqlite3.
Using a txt file as database is a terrible idea, go with SQL

Directly passing pandas data into zipline

I am currently looking for a way to directly pass in a pandas dataframe or csv file to zipline for simple backtesting WITHOUT having to ingest a data bundle. The reason is that I am planning to generate new data outside of the existing bundle during a backtest and it seems very inefficient to ingest a new bundle for every handle_data call.
I have been looking for this everywhere, including the source codes of zipline. I found that an older version of zipline has a 'data' param in the run_algo function call where you could pass in a df directly, but I can't find that old version at the moment. Is anyone attempting the same thing? Is there any way other than ingesting data bundles in the command line everytime?
I'm using zipline 1.3.0 and it actually does have a data param. This comment is from run_algo.py file of zipline:
data : pd.DataFrame, pd.Panel, or DataPortal, optional
The ohlcv data to run the backtest with.
This argument is mutually exclusive with:
``bundle``
``bundle_timestamp``
Hope it helped

what exactly is `xla_client` in the jax library?

If you read the jax source code you'll hit something called xla_client. Often imported like this
from . import xla_client
This implies that xla_client is a python module, but I can't find any file with that name or reference to a variable of that name.
I assume that it is related to https://pypi.org/project/jaxlib/, but this package just links back to the jax source code.
Can anybody clue me in?
The file you're referring to is stored at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/compiler/xla/python
Let me expound further: xla_client is partly a wrapper around a specially compiled c++ file called xla_extension.so, for example see
from . import xla_extension as _xla
and numerous references to _xla throughout xla_config. The source for this file is https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/xla/python/xla.cc, which we know because it says so quite clearly in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/xla/python/BUILD
pybind_extension(
name = "xla_extension",
srcs = [
"xla.cc",
],
...

Compiling and Llnking to used modules in external directory Compaq Fortran command prompt

I've already asked a similar question, here:
Linking to modules in external directory Compaq Visual Fortran command prompt
And I thought that the first answer was correct (that is, in the manual they say you can simply specify the path name before the module), but after deleting the temporary files in my library folder, this approach seemed to stop working. Trying with the /include[:path] approach, here is my .bat file:
df /include:..\FORTRAN_LIB\ __constants
myIO griddata_mod myfdgen myDiff magneticField /exe:magneticField
And an error is returned saying:
__constants
myIO
griddata_mod
myfdgen
myDiff
magneticField
f90: Severe: No such file or directory
... file is '__constants'
Again, I apologize that this question is VERY specific, but it seems like it should be simple and does not work at all.
p.s. Originally, I was using:
df ..\FORTRAN_LIB\__constants ..\FORTRAN_LIB\myIO
..\FORTRAN_LIB\griddata_mod ..\FORTRAN_LIB\myfdgen
..\FORTRAN_LIB\myDiff magneticField /exe:magneticField
But, as I've said, it stopped working after I deleted the temporary files in my FORTRAN_LIB folder. Also note, these .bat files used only one line, I've broken them into several lines just for readability. I would prefer using the /include[:path] option since that seems like a better solution.
Okay, so I think I figured out a workaround at the very least. I understood that the /include[:dir] specifies to search in "dir" for included files. But it seemed from documentation, that this also specifies to search for USEd modules but that doesn't seem to be the case.
My program now looks like this:
include '..\FORTRAN_LIB\__constants.f90'
include '..\FORTRAN_LIB\computeError.f90'
include '..\FORTRAN_LIB\griddata_mod.f90'
include '..\FORTRAN_LIB\myfdgen.f90'
include '..\FORTRAN_LIB\myDiff.f90'
include '..\FORTRAN_LIB\myIO.f90'
program magneticField
use constants
use computeError_mod
use griddata_mod
use myfdgen_mod
use myDiff_mod
use myIO_mod
implicit none
...
And my DF command like this:
df magneticField /exe:magneticField
And everything seems to work fine. It would be nicer to have the /include[:dir] option, but so long I'm able to reach in a separate directory, I'm satisfied. If anyone can find a better solution I'll switch the checkmark. I hope this helps with anyone else who was confused like me.