I'd like to store data from actual graphs. in other words we might the following for example:
paper: smith
finance type: outgoings
time | 0 10 20 30 ... etc
amount | 10 22 31 44 ... etc
I would like to store the variables paper, finance type and for each the graph data given by time-amount. there will be other variables also (note the above example is fictional)
I'm not here to get solutions although I hardly know anything about databases. Would like to get started. When I type in Google 'store data from graph in database' all I get is information about sql graph types, node etc. I need just some direction for the actual tools to use (MySql or another database type? XML?). I will eventually want to extract the graph data of person and use that information. Google is not being my friend at the moment and I don't know who to ask personally
The database wouldn't be that big but will eventually run into 1000s of entries.
It is possible to model this in a database, but if you hardly know anything about them, you should start learning a bit about ER schema's, normalization (just up to third normal form) and the basic DDL and DML queries.
Anyway, possible model with two tables:
TABLE 'graphs'
- ID
- paper
- finance type
TABLE 'graphdata'
- ID
- GRAPH_REF
- TIME
- AMOUNT
In your table graphs, you put 1 line for each graph you have. You might have a graph for 'smith, outgoings', one for 'smith, incomings', one for "deloitte, reports"... that would be three lines. The ID is just a counter.
In the table 'graphdata', you put 1 line for each data point. Again, the ID is just a counter. The GRAPH_REF is the ID of the graph in the 'graphs' table where this data-point belongs to.
So for your example, you'd have the following graphdata rows:
1 - 1 - 0 - 10
2 - 1 - 10 - 22
3 - 1 - 20 - 31
4 - 1 - 30 - 44
Are you following so far? Now you can make a webpage (or an application, anything you can program that can work with SQL - even Excel or Access will work) that gives a user the choice to create a new graph, or select an existing graph.
Creating a new graph would insert a new row in the 'graphs' table. Then, for each data point, you put a new row in the 'graphdata' table.
When they select an existing graph, you fetch the data points from the graph, and display to them. Maybe they can add/delete points?
Related
I am trying to build an automatic feature for a database that takes NOAA weather data and imports it into our own database tables.
Currently we have 3 steps:
1. Import the data literally into its own table to preserve the original data
2. Copy it's data into a table that better represents our own data in structure
3. Then convert that table into our own data
The problem I am having stems from the data that NOAA gives us. It comes in the following format:
Station Station_Name Elevation Latitude Longitude Date MXPN Measurement_Flag Quality_Flag Source_Flag Time_Of_Observation ...
Starting with MXPN (Maximum temperature for water in a pan) which for example is comprised of it's column and the 4 other columns after it, it repeats that same 5 columns for each form of weather observation. The problem though is that if a particular type of weather was not observed in any of the stations reported, that set of 5 columns will be completely omitted.
For example if you look at Central Florida stations, you will find no SNOW (Snowfall measured in mm). However, if you look at stations in New Jersey, you will find this column as they report snowfall. This means a 1:1 mapping of columns is not possible between different reports, and the order of columns may not be guaranteed.
Even worse, some of the weather types include wild cards in their definition, e.g. SN*# where * is a number from 0-8 representing the type of ground, and # is a number 1-7 representing the depth at which soil temperature was taken for the minimum soil temperature, and we'd like to collect these together.
All of these are column headers, and my instinct is to build a small Java program to map these properly to our data set as we'd like it. However, my superior believes it may be possible to have the database do this on a mass import, but he does not know how to do it.
Is there a way to do this as a mass import, or is it best for me to just write the Java program to convert the data to our format?
Systems in use:
MariaDB for the database.
Centos7 for the operating system (if it really becomes an issue)
Java is being done with JPA and Spring Boot, with hibernate where necessary.
You are creating a new table per each file.
I presume that the first 6 fields are always present, and that you have 0 or more occurrences of the next 5 fields. if you are using SQL Server i would approach it as follows,
Query the information_schema catalog to get a count of the fields in
the table. If the count= 6 then no observations are present, if 11
columns ,then you have 1 observation, if 17 then you have 2
observations, etc.
Now that you know the number of observations you can write some SQL
that will loop the over the observations and insert them into a
child table with a link back to a parent table which has the 1st 6
fields.
apologies if my assumptions are way off.
-HTH
I am using an external program that writes onto Access. This program collects data from an electronic form and writes all the data from the submitted form onto my Access database. However, the issue that has arisen and caused a lot of issues and slowed down our database, is that when it writes to my database, the data is not normalized.
The form looks something like this
Name: John Doe
DOb: April 1 1950
SIN: 123456789
Marital Status: Married
Phone: 123456789
Email: john#email.com
Then it writes everything on the form as one record using the Question as the field name and the entered data as the data. Something like this:
Name | DOB | SIN | Marital_Status | Phone | Email
John Doe| April 11 1950| 123456789| Married | 123456789| john#email.com
See this isn't much of an issue with the example form here, however, we have forms with about 100 questions which which we end up with a table with fields like:
Name|Date|Weather|Question1|Question2|Question3|Question4|...|Question100
.... and so forth.
As a noob, what I have done thus far was using the union sql query to manipulate the data so that it reads:
Name|Date|Weather|Question1
Name|Date|Weather|Question2
Name|Date|Weather|Question3
Name|Date|Weather|QuestionN
I have been able to get by with this but it is seriously slowing down my database and now I am having other issues.
How can I normalize this data when the external program writes data like this? I don't get to manipulate how the program writes to my Access Database.
Access 2010 has a feature called event-driven data macros, which are similar to triggers in other database systems. I don't personally have any experience using them, but it looks like you should be able to create an After Insert macro that will run when a new row is inserted. Within that macro you could split your questions up and insert them into a more normalized table (which you would then use to report off of).
You're doing it correctly, a union query is indeed the correct way to normalize a denormalized table. However, consider storing it normalized in addition to denormalized, so you can actually work with the data without having Access executing 100 queries every time you want to access your data. And consider splitting Name|Date|Weather to a different table, since you are repeating them 100 times per question.
You can store the union query result in a table by simply doing SELECT * INTO MyTable From UnionQuery. Combine the import from the other program with this query in a macro.
Obviously, this is not ideal. The ideal fix would be to manipulate the external program to not denormalize the data in the first place
I have a very large amount of data that would most naturally be represented as a tree:
Category 1
Sub-category 1
data point 1
attribute 1
Sub-cateogry 2
data point 1
attribute 1
attribute 2
data point 2
Category 2
Sub-category 1
Sub-category 1
data point 1
Sub-category 2
data point 1
data point 2
Sub-category 2
data point 1
data point 2
data point 3
...
The individual data points have text and numerical attributes, bit it doesn't really suited for representation as a set of related tables. I would like to be able to perform SQL-like queries, but I would also like to be able to browse through the data in a way that makes the tree structure of the data obvious, like with a file manager.
There's probably some class of application that is ideal for such a thing, but it isn't occurring to me at the moment. Some kind of combination of a database and a tree viewer control? Anyone know what it is I'm looking for? As always, I'm terrified of asking a question in the wrong forum, but I see some related questions here at stackoverflow, so hopefully it's OK. Thanks!
You could make a table like this
id
name
parent_id
This structure would allow for nested categories
You could then make a table that relates category and data points.
The java.swing packages contain several table and tree solutions such as the JTable and JTree classes. JTree can be easily constructed to produce the tree structure you are looking for (looks like a file directory.)
The JTable class can be used to create sortable and searchable tables, although you would have to borrow or write your own sort & search methods.
Although these are from Java, other languages offer similar structures that may serve your needs without using a database. That being said, "mySQL" is a very easy to use database and you can download the community DB package for free.
I have few tables as shown below
Polls
PollId Question Option
1 What 1
2 Why 4
Updates
UpdateId Text
1 Sleep
2 Play
Polls and updates are just two sample tables (In reality there are more tables like ,photos, videos,links etc). But when a user visit his home (like facebook new feed) he must be displayed with data relevant to him (no such data included in this example). ie I want to select data from all tables with less number of query executions. (ie, I want to present a mixture of datas, ie polls, photos, videos etc )
Currently, I'm fetching only ids and type (ie which table) from all of the tables and gather further data while iterating through this resultset. (ie from c# calling another SqlQuery) .
Is there a way to query the data from whole tables at once? (OUTER JOIN?, UNION?)
Or simply,
How can I select different type of entities at once in a single sql Query?
You could write your query so that you have one long select list for everything you want and it all comes back in one result set but I suspect that wouldn't work too well because you might have varying numbers of different types of items per user.
If you really must have it all in one hit then you can issue multiple queries in one go and get multiple result sets back. To handle this you can use an ADO.Net DataSet. See this SO example (but not the accepted answer - see Vikram Dibyal's answer as that gives a very basic overview of what I think you're asking for).
I won't copy and paste the stuff from the linked thread, just head over and take a look.
This is probably a dumb question, but I've hit a wall with this one at this current time.
I have some data which is hierarchial in nature which is in an ADO.NEt dataset. The first field is the ID, the second is the Name, the third is the Parent ID.
ID NAME Parent ID
1 Air Handling NULL
2 Compressor 1
3 Motor 4
4 Compressor 1
5 Motor 2
6 Controller 4
7 Controller 2
So the tree would look like the following:
1- Air Handling
4- Compressor
6 - Controller
3 - Motor
2- Compressor
7- Controller
5 - Motor
What I'm trying to figure our is how to get the dataset in the same order that ths would be viewed in a treeview, which in this case is the levels at the appropriate levels for the nodes and then the children at the appropriate levels sorted by the name.
It would be like binding this to a treeview and then simply working your way down the nodes to get the right order.
Any links or direction would be greatly appreciated.
I would re-organize the data to be more normalized, however if you must work with what you have, do nested for statements (where the first one looks for the null in the parentID field. If you don't know how many layers deep the data goes, it will make your job more difficult, but it can be done.