I have a question. In an application close to 500 views have been designed based on different reportable granularities. My concern is will this high number impact performance of the application in any way. One more point , the views are interlinked. So all of the views are accessed at some point of time. Could you please provide insights on this.
Thanks And Regards
Aritra Bhattacharya
Related
Many people are telling me that Nebula NatTable is better for handling large amount of data. When I ask how it is better no one is giving me correct answer. Could anyone please tell me why NatTable is best for handling large amount of data? How does it differ from other tables in handling large amount of data?
NatTable is a virtual table by design. This means for rendering it only processes the data that is visible to the user. Especially with regards to a large number of columns this is a big difference to other table implementations. Additionally the usage of GlazedLists brings performance optimizations.
If you have a look at our getting started tutorial you probably find some additional information I forgot.
https://www.vogella.com/tutorials/NatTable/article.html
I'm currently developing a game as a personal project! One of the main aspects of the game is the map - it will be a multiplayer game whereby players can capture different areas of a map by building things.
The map will be very big! Around 1000x1000 (so 1M tiles). The game will also have a fair amount of players on the map at the same time (100-1000), who will be constantly capturing new areas and stealing areas from other players, as such the tile database will be constantly changing in real time.
My question is - does anyone have any recommendations on how to go about this? My initial ideas were:
Have a MongoDB database with a collection of tiles.
Pros: Can query for certain areas of the map so that the client only has to download a portion of the map every time
Cons: Collection will be very large (several GB) (each tile would need to have X, Y coordinates, a resource level, an owner, and whether another player is contesting the tile)
Have an SQL database
Pros: Will be lighter in size, probably quicker to query.
Cons: Might not be able to be written to and edited easily in real time.
Any thoughts / direction would be greatly appreciated!
Thank you!
If I understand the question, It sound like spatial indexing is the way to go. with a good spatial index it will be trivial to locate the player and determine which parts of the map are nearby. I've only ever used it for geo-data, but with the correct polygons it should be usable in your scenario as well.
Microsoft does a much better explanation than I can give in a stackoverflow answer, and similar functionality exists in MongoDB. Hope that helps.
A million tiles is not necessarily a large quantity of objects for a database to manage. Like other types of addressable assets (e.g. airline seats, hotel rooms, concert tickets), each map tile will have a primary key identifier that is indexed for fast retrieval and precise, targeted updates.
Depending on the rules regarding movement across tiles and how much volatile information is involved in rendering a tile, you may want to devise a prefetching scheme that anticipates which tiles a player might need next and downloads them in advance to minimize delays.
In order for your application to accommodate hundreds or thousands of users who are simultaneously viewing, modifying, and taking ownership of specific records without suffering from lock timeouts and deadlocks, your database model and query workload will need to be designed for concurrency. SQL-based databases allow you to use normalization techniques to arrive at a data model which not only accurately represents the data you're managing, but also eliminates the risk of duplicate records, double-booking, lost updates, and other anomalies. If your data model is adequately normalized and your application is making proper use of atomic transactions (units of work), the A.C.I.D. properties of SQL-based databases offer powerful, built-in protection for your data with minimal application coding.
I'm working on a new project right now and thinking of using an ORM beyond that of OpenAccess or LLBLGen Pro or Subsonic.This project may have great quantities and hits concurrent,So our performance requirements is very high.
Please compare and recommend it to me.
Thanks
Jim.
Jim,
For the best results in answering this question, you'll need to do your own comparison since your specific requirements and data access scenarios will likely affect the results of any such performance testing.
That said, we use LLBLGen for a high throughput web application and the performance is exceptional. We have found that the big issue is in the application design itself. Using SQL Server Profiler we are able to see (during development) which parts of the application create a lot of hits on the database. The biggest penalty we found was with loading a grid and then doing another database operation OnDataBinding / DataBound events. This creates a huge amount of traffic to the SQL Server database, a lot of reads and a lot of disk swapping. We have been very well served by making sure we get all the data in the first query by making a good design choice when building the set of data/joins/etc. when building the application -- or refactoring it later once we find the performance is slow.
The overhead for LLBLGen, at least, is very minimal. It's fast even when creating huge numbers of objects. The much, much bigger performance hit comes when we make queries that spawn other queries (example above) and our DB reads go through the roof.
You may wish to evaluate both for which one you feel is a better match for your skills and productivity as well.
Im new to database design and need some guidance.
A lot of new data is inserted to my database throughout the day. (100k rows per day)
The data is never modified or deleted once it has been inserted.
How can I optimize this database for retrieval speed?
My ideas
Create two databases (and possible on different hard drives) and merge the two at night when traffic is low
Create some special indexes...
Your recommendation is highly appreciated.
UPDATE:
My database only has a single table.
100k/day is actually fairly low. 3M/month, 40M/year. You can store 10 years archive and not reach 1B rows.
The most important thing to choose in your design will be the clustered key(s). You need to make sure that they are narrow and can serve all the queries your application will normally use. Any query that will end up in table scan will completely trash your memory by fetching in the entire table. So, no surprises there, your driving factor in your design is the actual load you'll have: exactly what queries will you be running.
A common problem (more often neglected than not) with any high insert rate is that eventually every row inserted will have to be deleted. Not acknowledging this is a pipe dream. The proper strategy depends on many factors, but probably the best bet is on a sliding window partitioning scheme. See How to Implement an Automatic Sliding Window in a Partitioned Table. This cannot be some afterthought, the choice for how to remove data will permeate every aspect of your design and you better start making a strategy now.
The best tip I can give which all big sites use to speed up there website is:
CACHE CACHE CACHE
use redis/memcached to cache your data! Because memory is (blazingly)fast and disc I/O is expensive.
Queue writes
Also for extra performance you could queue up the writes in memory for a little while before flushing them to disc -> writting them to SQL database. Off course then you have the risk off losing data if you keep it in memory and your computer crashes or has power failure or something
Context missing
Also I don't think you gave us much context!
What I think is missing is:
architecture.
What kind of server are you having VPS/shared hosting.
What kind of Operating system does it have linux/windows/macosx
computer specifics like how much memory available, cpu etc.
a find your definition of data a bit vague. Could you not attach a diagram or something which explains your domain a little bit. For example something like
this using http://yuml.me/
Your requirements are way to general. For MS SQL server 100k (more or less "normal") records per days should not be a problem, if you have decent hardware. Obviously you want to write fast to the database, but you ask for optimization for retrieval performance. That does not match very well! ;-) Tuning a database is a special skill on its own. So you will never get the general answer you would like to have.
We're adding extra login information to an existing database record on the order of 3.85KB per login.
There are two concerns about this:
1) Is this too much on-the-wire data added per login?
2) Is this too much extra data we're storing in the database per login?
Given todays technology, are these valid concerns?
Background:
We don't have concrete usage figures, but we average about 5,000 logins per month. We hope to scale to larger customers, howerver, still in the 10's of 1000's per month, not 1000's per second.
In the US (our market) broadband has 60% market adoption.
Assuming you have ~80,000 logins per month, you would be adding ~ 3.75 GB per YEAR to your database table.
If you are using a decent RDBMS like MySQL, PostgreSQL, SQLServer, Oracle, etc... this is a laughable amount of data and traffic. After several years, you might want to start looking at archiving some of it. But by then, who knows what the application will look like?
It's always important to consider how you are going to be querying this data, so that you don't run into performance bottlenecks. Without those details, I cannot comment very usefully on that aspect.
But to answer your concern, do not be concerned. Just always keep thinking ahead.
How many users do you have? How often do they have to log in? Are they likely to be on fast connections, or damp pieces of string? Do you mean you're really adding 3.85K per time someone logs in, or per user account? How long do you have to store the data? What benefit does it give you? How does it compare with the amount of data you're already storing? (i.e. is most of your data going to be due to this new part, or will it be a drop in the ocean?)
In short - this is a very context-sensitive question :)
Given that storage and hardware are SOOO cheap these days (relatively speaking of course) this should not be a concern. Obviously if you need the data then you need the data! You can use replication to several locations so that the added data doesn't need to move over the wire as far (such as a server on the west coast and the east coast). You can manage your data by separating it by state to minimize the size of your tables (similar to what banks do, choose state as part of the login process so that they look to the right data store). You can use horizontal partitioning to minimize the number or records per table to keep your queries speedy. Lots of ways to keep large data optimized. Also check into Lucene if you plan to do lots of reads to this data.
In terms of today's average server technology it's not a problem. In terms of your server technology it could be a problem. You need to provide more info.
In terms of storage, this is peanuts, although you want to eventually archive or throw out old data.
In terms of network (?) traffic, this is not much on the server end, but it will affect the speed at which your website appears to load and function for a good portion of customers. Although many have broadband, someone somewhere will try it on edge or modem or while using bit torrent heavily, your site will appear slow or malfunction altogether and you'll get loud complaints all over the web. Does it matter? If your users really need your service, they can surely wait, if you are developing new twitter the page load time increase is hardly acceptable.