I am working in an application that handles different types of currencies. The application takes the currencies as input and converts them into USD and stores it in the database. Some data processing need to happen which creates issues with inconsistent rounding. I would like to find out the best way to handle store currencies. Is it better to store them in the native currency or store them all in one type like how the application is doing right now and display them in different currencies based on the culture?
I would have a table
currencies(id, name, sign)
and I would make sure that there are proper conversions:
conversions(id, from_currency, to_currency, value)
You might also want to have a table with conversion histories.
Related
I'm currently retrieving data from a vehicle fleet (fuel used, distance traveled...) through the manufacturer API. For each set of data, there is the date when the metric has been mesured. I will retrieve the data everyday through a rcon call and store it in my DB. The purpose is to keep the history for each vehicle so that I could get every metrics mesured for a vehicle between X date and X date later on.
I first thought of this :
But it doesn't seem right because of the "1,1" cardinality, so i transformed the 3 way relationship into 2 normal relationship :
At this point i wondered wether i could not simply store the field date in the metric entity (because I noticed the API would give me a datetime, so it's very unlikely that two metrics will be mesured at the same time):
And finally, i was wondering if putting everything in a data entity would not even be easier (but it feels kinda wrong):
So i'm completely lost as to what would be the best way to do this. Could someone tell me which way is the best or even if there is a better way and why ?
In my app, I'm just using a SQLlite database for development. Now in the migration, I declare a DATE datatype which laravel seems to handle without any problem, and in the database itself creates it as a varchar.
According to this nice article (http://www.sqlitetutorial.net/sqlite-date/) SQLite has basically got three options for handling dates:
Using the TEXT storage class for storing SQLite date and time Using
REAL storage class to store SQLite date and time values
Using INTEGER to store SQLite date and time values
So as I'm trying to formulate my approach, I'm thinking ahead that I will likely end up, at some point, need to step up and move to a higher performance SQL database (mySQL / Postgres / etc. ) And then may have datatype translation challenges.
But then also, at the application layer, Laravel itself has some manipulations.
Now, the question I'm asking is this, What is the benefit of one type over another? Is there some kind of reason to choose one type over another? My thinking is that TEXT is nice and human-readable for backend support, but it may require addiotnal coding to manipulate strings.
INTEGERS are probably more efficient, and would be translatable to a bigger SQL server easier than text.
Does anyone know of a comparison of the pro's and con's of various choices?
Any advice? Thanks in advance.
The size of integer is 4 bytes. The size of a letter in text is 1 byte.
To represent date and time you need 1 UTC number when you use integer. So its much better to user 4 bytes of integer than using 8 bytes of text. I dont see how real can be better than integer for the exact same reason. I would say you should use integer.
I'm creating an app that has to store historical financial data for various stocks.
I currently have a stock table where the columns are stock symbol, stockname along with numerical data which I'm trying to decide how to store.
For example, for the column stockprice, I want to store an entire hash where the key is the date as a string and the value is the stock price. This information should be easily accessible(fast random access). I've read a bit about serializing, but I wonder if this is the best option(or if it's even applicable at all). Is there a way to instead automatically generate an sqlite table for each stock entered and create columns representing the date and rows representing the stockprice?
I appreciate all insight into this matter and perhaps some clarification on whether this is exactly where I should use serialization or whether there is a better alternative
EDIT 1: Is ActiveModel Serialization relevant? (http://railscasts.com/episodes/409-active-model-serializers)
EDIT 2: Is it advisable to consider instead creating a Stockprice model & table where the model belongs_to a stock and a stock has_many stockprices. THe stockprice table would have a regular id, a stock_id(for which it belongs) to and a date column and a stockprice value column. I'd appreciate some analysis on the run-time memory-time usage of this in comparison to serialization and how to analyze it in the future
You are correct, it is possible to store it as a hash. I don't have any metrics for serialize but I would suggest starting this way and optimizing your data storage later if you begin to notice a significant impact on your application.
You're migration would look something like this (be sure to use the text data type):
def self.up
add_column :stocks, :price, :text
end
In your model you will need to add
serialize :price
You will be able to create price as a hash and store it directly.
stock = Stock.new
stock.price = { :date => "#{Time.now}", :amount => 25.2 }
stock.save
EDIT: I would start with the serialization unless you have designed functionality that is specific to stock_price. Since the convention in Rails is to have a model per each table, you would end up with a class for stock_price. It isn't necessary to dedicate a class for stock_price unless you have specific methods in mind for that class. Also, depending on your design, you may be able to keep the stocks class more cohesive by keeping the stock price as an attribute of stocks.
you mentioned 'This information should be easily accessible(fast random access)' - in which case serialized column is not a good option. Lets say you keep 20 years of data, then it would be 20*365 key value pairs in the serialized price. But you might be interested in only a subset of this for an usecase - say plotting last 6 months. If you go with the serialize option, that entire data (for price field) will be transported from db to ruby process and gets de-serialized.Then again you need to filter the price hash in ruby process. Where as in case of a seperate table for price, db can do the filtering for you and you can have a fast response with good indices.
Have you explored any time series dbs?
Is there a list of best practice MySQL data types for common applications. For example, the list would contain the best data type and size for id, ip address, email, subject, summary, description content, url, date (timestamp and human readable), geo points, media height, media width, media duration, etc
Thank you!!!
i don't know of any, so let's start one!
numeric ID/auto_increment primary keys: use an unsigned integer. do not use 0 as a value. and keep in mind the maximum value of of the various sizes, i.e. don't use int if you don't need 4 billion values when the 16 million offered by mediumint will suffice.
dates: unless you specifically need dates/times that are outside the supported range of mysql's DATE and TIME types, use them! if you instead use unix timestamps, you have to convert them to use the built-in date and time functions. if your app needs unix timestamps, you can always convert the standard date and time data types on the way out using unix_timestamp().
ip addresses: use inet_aton() and inet_ntoa() since it easily compacts an ip address in to 4 bytes and gives you the ability to do range searches that utilize indexes.
Integer Display Width You likely define your integers something like this "INT(4)" but have been baffled by the fact that (4) has no real effect on the stored numbers. In other words, you can store numbers like 999999 just fine. The reason is that for integers, (4) is the display width, and only has an effect if used with the ZEROFILL modifier. Further, this is for display purposes only, so you could define a column as "INT(4) ZEROFILL" and store 99999. If you stored 999, the mysql REPL (console) would output 0999 when you've selected this column.
In other words, if you don't need the ZEROFILL stuff, you can leave off the display width.
Money: Use the Decimal data type. Based on real-world production scenarios I recommend (19,8).
EDIT: My original recommendation was (19,4); however, I've recently run into a production issue where the client reported that they absolutely needed decimal with a "scale" of "8"; thus "4" wasn't enough and was causing improper tax calculations. I now recommend (19,8) based on a real-world scenario. I would love to hear stories needing a more granular scale.
What are your thoughts on this? I'm working on integrating some new data that's in a tab-delimited text format, and all of the decimal columns are kept as single integers; in order to determine the decimal amount you need to multiply the number by .01. It does this for things like percentages, weight and pricing information. For example, an item's price is expressed as 3259 in the data files, and when I want to display it I would need to multiply it in order to get the "real" amount of 32.59.
Do you think this is a good or bad idea? Should I be keeping my data structure identical to the one provided by the vendor, or should I make the database columns true decimals and use SSIS or some sort of ETL process to automatically multiply the integer columns into their decimal equivalent? At this point I haven't decided if I am going to use an ORM or Stored Procedures or what to retrieve the data, so I'm trying to think long term and decide which approach to use. I could also easily just handle this in code from a DTO or similar, something along the lines of:
public class Product
{
// ...
private int _price;
public decimal Price
{
get
{
return (this._price * .01);
}
set
{
this._price = (value / .01);
}
}
}
But that seems like extra and unnecessary work on the part of a class. How would you approach this, keeping in mind that the data is provided in the integer format by a vendor that you will regularly need to get updates from.
"Do you think this is a good or bad idea?"
Bad.
"Should I be keeping my data structure identical to the one provided by the vendor?"
No.
"Should I make the database columns true decimals?"
Yes.
It's so much simpler to do what's right. Currently, the data is transmitted with no "." to separate the whole numbers from the decimals; that doesn't any real significance.
The data is decimal. Decimal math works. Use the decimal math provided by your language and database. Don't invent your own version of Decimal arithmetic.
Personally I would much prefer to have the data stored correctly in my database and just do a simple conversion every time an update comes in.
Pedantically: they aren't kept as ints either. They are strings that require parsing.
Philisophically: you have information in the file and you should write data into the database. That means transforming the information in any ways necessary to make it meaningful/useful. If you don't do this transform up front, then you'll be doomed to repeat the transform across all consumers of the database.
There are some scenarios where you aren't allowed to transform the data, such as being able to answer the question: "What was in the file?". Those scenarios would require the data to be written as string - if the parse failed, you wouldn't have an accurate representation of the file.
In my mind the most important facet of using Decimal over Int in this scenario is maintainability.
Data stored in the tables should be clearly meaningful without need for arbitrary manipulation. If manipulation is required is should be clearly evident that it is (such as from the field name).
I recently dealt with data where days of the week were stored as values 2-8. You can not imagine the fall out this caused (testing didn't show the problem for a variety of reasons, but live use did cause political explosions).
If you do ever run in to such a situation, I would be absolutely certain to ensure data can not be written to or read from the table without use of stored procedures or views. This enables you to ensure the necessary manipulation is both enforced and documented. If you don't have both of these, some poor sod who follows you in the future will curse your very name.