What is the difference between Sequential and Paralle Process Order in Analysis Services Process Task, in SSIS ?
Here's an article that answers your question. Parallel processing allows the server to process cubes, measure groups, and partitions in parallel. This requires spinning off threads on separate processors which can impact general server performance. If your server is already hurting when you process changes to the cube, then consider using Sequential.
Related
I noticed that my queries are running faster on my local machine compare to my server because on both machines only one core of the CPU is being used. Is there a way to enable multi-threading so I can use 12 (or all 24 cores) instead of just one?
I didn't find anything in the documentation to set this up but saw that other graph databases do support it. If it is supported by default, what could cause it to only use a single core?
GraphDB by default will load all available CPU cores unless limited by the license type. The Free Edition has a limitation up to 2 concurrent read operations. However, I suspect that what you ask for is how to enable the query parallelism (decompose the query into smaller tasks and execute them in parallel).
Write operations in GDB SE/EE will always be split into multiple parallel tasks, so you will benefit from the multiple cores. GraphDB Free is limited to a single core due to commercial reasons.
Read operations are always executed on a single thread because in the general case the queries run faster. In some specific scenarios like heavy aggregates over large collections parallelizing the query execution may have substantial benefit, but this is currently not supported.
So to sum up having multiple cores will help you only handle more concurrent queries, but not process them faster. This design limitation may change in the upcoming versions.
I am currently working on a project to improve cube processing time. The cube currently consists of 50 facts and 160 dimensions and it takes about 4 hours to process the cube. What would be the best way to benchmark cube processing performance before embarking on troubleshooting bottlenecks. The largest dimension consists of about nine million records while the largest fact table consists of about 250 million records. How would you go about finding bottlenecks and what parameters would influence the processing time the most. Any help is highly appreciated.
Having done a lot of SSAS processing optimization I would start with some monitoring. First setup some performance counters to monitor available memory, disk, CPU and network on the SSAS server and the database server. Some good perfmon counters (and recommendations about baselining processing performance) are in section 4.1.1 in the SSAS performance guide.
Second I would start a profiler trace connected to SSAS with the default events. Then when done processing Save As... Trace Table in profiler. Then look for the longest duration events in the SQL table you save it to. Then you know where to spend your time optimizing.
Feel free to write back with your specific longest duration events if you need help. Please also specify exactly how you are processing (like ProcessFull on the database or something else).
We have setup a 3 node performance cluster with 16G RAM and 8 Cores each. Our use case is to write 1 million rows to a single table with 101 columns which is currently taking 57-58 mins for the write operation. What should be our first steps towards optimizing the write performance on our cluster?
The first thing I would do is look at the application that is performing the writes:
What language is the application written in and what driver is it using? Some drivers can offer better inherent performance than others. i.e. Python, Ruby, and Node.js drivers may only make use of one thread, so running multiple instances of your application (1 per core) may be something to consider. Your question is tagged 'spark-cassandra-connector' so that possibly indicates your are using that, which uses the datastax java driver, which should perform well as a single instance.
Are your writes asynchronous or are you writing data one at a time? How many writes does it execute concurrently? Too many concurrent writes could cause pressure in Cassandra, but not very many concurrent writes could reduce throughput. If you are using the spark connector are you using saveToCassandra/saveAsCassandraTable or something else?
Are you using batching? If you are, how many rows are you inserting/updating per batch? Too many rows could put a lot of pressure on cassandra. Additionally, are all of your inserts/updates going to the same partition within a batch? If they aren't in the same partition, you should consider batching them up.
Spark Connector Specific: You can tune the write settings, like batch size, batch level (i.e. by partition or by replica set), write throughput in mb per core, etc. You can see all these settings here.
The second thing I would look at is look at metrics on the cassandra side on each individual node.
What does the garbage collection metrics look like? You can enable GC logs by uncommenting lines in conf/cassandra-env.sh (As shown here). Are Your Garbage Collection Logs Speaking to You?. You may need to tune your GC settings, if you are using an 8GB heap the defaults are usually pretty good.
Do your cpu and disk utilization indicate that your systems are under heavy load? Your hardware or configuration could be constraining your capability Selecting hardware for enterprise implementations
Commands like nodetool cfhistograms and nodetool proxyhistograms will help you understand how long your requests are taking (proxyhistograms) and cfhistograms (latencies in particular) could give you insight into any other possibile disparities between how long it takes to process the request vs. perform mutation operations.
I am aware that multiple connections use multiple CPU cores in postgres and hence run in parallel.But when I execute a long running query say 30 seconds(Let's assume that this cannot be optimized further), the I/O is blocked and it does not run any other query from the same client/connection.
Is this by design or can it be improved ?
So I am assuming that the best way to run long running queries is to get a new connection or not to run any other query in the same connection until that query is complete ?
It is a design limitation.
PostgreSQL uses one process per connection, and has one session per process. Each process is single-threaded and makes heavy use of globals inherited via fork() from the postmaster. Shared memory is managed explicitly.
This has some big advantages in ease of development, debugging and maintenance, and makes the system more robust in the face of errors. However, it makes it significantly harder to add parallelization on a query level.
There's ongoing work to add parallel query support, but at present the system is really limited to using one CPU core per query. It can benefit from parallel I/O in some areas, like bitmap index scans (via effective_io_concurrency), but not in others.
There are some IMO pretty hacky workarounds like PL/Proxy but mostly you have to deal with parallelization yourself client-side if it's needed. This is rapidly becoming one of the more significant limitations impacting PostgreSQL. Applications can split up large queries into multiple smaller queries that affect a subset of the data, then unify client-side (or into an unlogged table that then gets further processed), i.e. a map/reduce-style pattern. If a mix of big long running queries and low-latency OLTP queries is needed, multiple connections are required and the app should usually use an internal connection pool.
I am using MySQL database and I would like to know if I make multiple (500 and more) queries simultaneously in order to get information from multiple tables, how these queries are handled? Sequentially or in parallel?
Queries are always handled in parallel between multiple sessions (i.e. client connections). All queries on a single connections are run one-after-another. The level of parallelism between multiple connections can be configured depending on your available server resources.
Generally, some operations are guarded between individual query sessions (called transactions). These are supported by InnoDB backends, but not MyISAM tables (but it supports a concept called atomic operations). There are various level of isolation which differ in which operations are guarded from each other (and thus how operations in one parallel transactions affect another) and in their performance impact.
For more information read about transactions in general and the implementation in MySQL.
Each connection can run a maximum of one query at once, and does it in a single thread. The server opens one thread per query.
Normally, hopefully, queries don't block each other. Depending on the engine and the queries however, they may. There is a lot of locking in MySQL which is discussed in some detail in the manual.
However, if they don't block each other, they can still slow each other down by consuming resources. IO operations are a particular source of these slow-downs. If your data don't fit in memory, you should really limit the number of parallel queries to what your IO subsystem can handle, or things will get really bad. Measurement is the key.
I would normally say that if 500 queries are running at once (and NOT waiting on locks), you may not be getting best value from your hardware (do you have 500 cores? How many threads are waiting for IO?)
Normally all queries will be run in parallel.
But... there are some exceptions to that. Depending on your transaction isolation level a row can be locked while updating. Read more about that over here: http://dev.mysql.com/doc/refman/5.1/en/dynindex-isolevel.html