I am trying to use python to read a .cdpg file. It was generated by the labview code. I do not have access to any information about the structure of the file. Using another post I have had some success, but the numbers are not making any sense. I do not know if my code is wrong or if my interpretation of the data is wrong.
The code I am using is:
import struct
with open(file, mode='rb') as file: # b is important -> binary
fileContent = file.read()
ints = struct.unpack("i" * ((len(fileContent) -24) // 4), fileContent[20:-4])
print(ints)
The file is located here. Any guidance would be greatly appreciated.
Thank you,
T
According to the documentation here https://www.ni.com/pl-pl/support/documentation/supplemental/12/logging-data-with-national-instruments-citadel.html
The .cdpg files contain trace data. Citadel stores data in a
compressed format; therefore, you cannot read and extract data from
these files directly. You must use the Citadel API in the DSC Module
or the Historical Data Viewer to access trace data. Refer to the
Citadel Operations section for more information about retrieving data
from a Citadel database.
.cdpg is a closed format containing compressed data. You won't be able to interpret them properly not knowing the file format structure. You can read the raw binary content and this is what you're actually doing with your example Python code
Related
I have multiple ecospold (version 1) files with LCI data that I want to convert to a custom excel format. I need all data given in the ecospold file. For my own convinience I want to use python to complete this task.
My research until now has lead me to the following conclusions:
There exist at least two converters (by GLAD and openLCA) to convert ecospold formats (1 and 2) to e.g. the ILCD. But those formats are not helping me to go anywhere, since I need to have all the data accessible in python and in order to then write it into my custom excel format.
To get the data in python, the package pyecospold (https://github.com/sami-m-g/pyecospold) seems to be a suitable choice.
According to the README that can be found at the pyecospold github repository,
ecoSpold = parse_file("data/v1/v1_1.xml") # Replace with your own XML file
should do the job. So I implemented the following lines:
import os
from pyecospold import parse_file, save_file, Defaults
from lxml import etree
cd = os.getcwd()
path_input = cd + r'\inputs\ecospold_test.xml'
# Parse the required XML file to EcoSpold class.
es = parse_file('inputs/ecospold_test.xml')
Now I run into the error:
TypeError: parse_file() missing 2 required positional arguments: 'schema_path' and 'ecospold_lookup'
I understood that a schema in xsd format is needed, therefore I got the schema files from the github and amended my last line of code:
es = parse_file('inputs/ecospold_test.xml', 'inputs/schemas/v1/EcoSpold01Dataset.xsd')
Now there is still one argument missing:
TypeError: parse_file() missing 1 required positional argument: 'ecospold_lookup'
Since I have no experience in parsing xml files in python, I have no idea what to do with this. Additionally, I am confused why the README does not say anything about those additionally needed arguments.
My second idea was to use brightway to get the data into python. But since brightway itself is quite an extensive package, I could not find a simple (or any) way to do this. (Sadly, the notebooks linked in the answer of this question Import Ecoinvent 2.2 Ecospold files into Brightway do not exist anymore)
Another option would of course be to write my own parser. But because I am lacking experience and pyecospold does exactly this (at least in my understanding), I would like to avoid this option.
Additionally, there in openLCA it is possible to read in ecospold files and then export them to an excel format. From this excel format I could of course make my custom excel format. The problem here is that I have no idea how to automize this, because I do not want to read in and export each file individually and manually in openLCA.
If anyone has an idea on how to solve one of my subproblems or a good alternative on how to solve my general problem, I would be very thankful. :)
So, I created a minigame bot on telegram. The bot just contains a fishing game, and it's already running. I want if a user fishes and gets a fish, the fish will be stored in a database. So the user can see what he got while fishing. Does this is require SQL?
I haven't tried anything, because I don't understand about storing data in python. If there is a tutorial related to this, please share it in the comments. Thank you
You can use anything to store user data, including text files.
The simplest approaches to storing data can be serializing a dictionary to JSON with the builtin json module:
DATABASE = 'database.json' # (name or extension don't actually matter)
import json
# loading
with open(DATABASE, 'r', encoding='utf-8') as fd:
user_data = json.load(fd)
user_data[1234] = 5 # pretend user 1234 scored 5 points
# saving
with open(DATABASE, 'w', encoding='utf-8') as fd:
json.dump(user_data, fd)
This would only support simple data-types. If you need to store custom classes, as long as you don't upgrade your Python version, you can use the built-in pickle module:
DATABASE = 'database.pickle' # (name or extension don't actually matter)
import pickle
# loading
with open(DATABASE, 'rb') as fd:
user_data = pickle.load(fd)
user_data[1234] = 5 # pretend user 1234 scored 5 points
# saving
with open(DATABASE, 'wb') as fd:
pickle.dump(user_data, fd)
Whether this is a good idea or not depends on how many users you expect your bot to have. If it's even a hundred, these approaches will work just fine. If it's in the thousands, perhaps you could use separate files per user, and still be okay. If it's more than that, then yes, using any database, including the built-in sqlite3 module, would be a better idea. There are many modules for different database engines, but using SQLite is often enough (and there are also libraries that make using SQLite easier).
Telethon itself uses the sqlite3 module to store the authorization key, and a cache for users it has seen. It's not recommended to reuse that same file for your own needs though. It's better to create your own database file if you choose to use sqlite3.
Using a txt file as database is a terrible idea, go with SQL
This is my data flow for my system:
Because i can not found a demo to config a raw16, and i did not found the enum type "enum CyU3PMipicsiDataFormat_t " which not contain a RAW16type,
so i did't known how to transfer my raw16 data to the host.
I try to use the yuv422 configuration to transfer my raw data to the host, and i really received data from the CX3 by e-cam, but the image is wrong for the e-cam use the yuv2 formating to resolve the raw data. And now I think i can use the matlab to grap a frame and deal with it. But when i use the matlab getting a snashot and i found the data is a
type like this: 1280*800*3(full frame size:1280x800). Is it the matlab regard as a yuv data? and how can i config the cx3 to support raw16 or how to deal with the data i grap from the cx3 with the yuv format transfer.
Is there any other developer meet the requirement like me?
I have a sensor that can give me a message. I want to transfer this data via can. My receiver wants me from a .dbc file. (Database for can). In my local(with PCAN) i can send a data in 1 miliseceond.
But after i put the can cable to my receiver it takes a data in 1 second. I think the problem is in .dbc file. Is there any definition to config data rate in .dbc file ?
I am having few ISO8583 logs in a text file. I want to parse these logs from this text file and write them to any database with some descriptive information such as class of the message, message function, message origin, processing code, response code etc.
I am new to the BASE24/ISO8583 and was trying to find any ready-made parser for this. Is there any such parser available ? Does jPOS provides such functionality ?
EDIT
I have the logs in ISO8583 format in ".log" file as given below:
MTI : 0200
Field-3 : 201234
Field-4 : 000000010000
Field-7 : 0110722180
Field-11 : 123456
Field-44 : A5DFGR
Field-105 : ABCDEFGHIJ 1234567890
This is same as the format given in the link shared by you.
It also consists of hex dump but I dont want to parse that.
The code given in the link is doing packing and unpacking of the message where as what I am trying is to read these logs (in unpacked form) and write them into a database table.
I think i need to write my own code for this and use the jPOS packagers in it.
It really depends on the format of the log file - are the ISO8583 messages - HexStrings, and HexDump an XML representation of ISO8583, some other application trace file ?
Once you know the format and it might require some massaging - you will want to research the ISOMsg.unpack() methods using the appropriate jPOS packager. the packager defines the field structure - of the various ISO8583 fields and field construction (lengths, character set, etc.)
a good example was found at the following blog post: looking at the "Parse (unpack) ISO Message" seciton http://jimmod.com/blog/2011/07/26/jimmys-blog-iso-8583-tutorial-build-and-parse-iso-message-using-jpos-library/
You mention - Base24 - jPOS does have a few packagers that might be close starting point.:
https://github.com/jpos/jPOS/blob/master/jpos/src/dist/cfg/packager/base24.xml
Those human-readable log formats are usually difficult to parse without loosing information. Moreover, the logs are probably PCI compliant so there's a lot of masked information there. You want to ask for ah hex dump of the messages.
what is displayed in log file is parsed ISO.Hence you need not use jpos.jpos is only for packing and unpacking when you transmit the message.
Assign the field to variable and write in DB
for example,Field 39 is response code.
Using jpos is good idea. You should go for your custom packager design class.