I am trying to load a dataset to GraphDB 7.0. I wrote a Python script to transform and load the data on Sublime Text 3. The program suddenly stopped working and closed, the computer threatened to restart but didn't, and I lost several hours worth of computing as GraphDB doesn't let me query the inserts. This is the error I get on GraphDB:
The currently selected repository cannot be used for queries due to an error:
org.openrdf.repository.RepositoryException: java.lang.RuntimeException: There is not enough memory for the entity pool to load: 65728645 bytes are required but there are 0 left. Maybe cache-memory/tuple-index-memory is too big.
I set the JVM as follows:
-Xms8g
-Xmx9g
I don't exactly remember what I set as the values for the cache and index memories. How do I resolve this issue?
For the record, the database I need to parse has about 300k records. The program shut shop at about 50k. What do I need to do to resolve this issue?
Open the workbench and check the amount of memory you have given to cache memory.
Xmx should be a value that is enough for
cache-memory + memory-for-queries + entity-pool-hash-memory
sadly the latter cannot be calculated easily because it depends on the number of entities in the repository. You will either have to:
Increase the java memory with a bigger value for Xmx
Decrease the value for cache memory
Related
I need help, maybe someone had similar problem with JMeter report generation
I have JMeter script with SOAP API requests, which are placing a purchasing order. There are no issues during order creation time, but when all requests are finished and JMeter is trying to generate report I am getting an error:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid8676.hprof ...
Heap dump file created [7011161840 bytes in 93.212 secs]
Uncaught Exception java.lang.OutOfMemoryError: Java heap space in thread Thread[StandardJMeterEngine,5,main]. See log file for details.
I used JConsole to monitor JMeter during execution and noticed that heap mostly was at 25% during test run and went up to 100% during report generation.
Thanks in advance
I've hit this issue numerous times when the results file from a test is very large. What you need to do is set the max heap size before generating the Dashboard report. What value you can set the max heap size depends on whether you're running on a 32-bit OS versus a 64-bit and how much RAM is available on the machine/VM. If the machine you're generating the report on doesn't have enough RAM available, then you can copy the results file to another machine which has more. For example, when I hit this issue on a test VM, I normally just copy the results file locally to my laptop which has 16GB and run the following command:
JVM_ARGS="-Xms3072m -Xmx12g" && export JVM_ARGS && jmeter -g resultsFile.jtl -o outputReport
The error clearly states that JMeter lacks Java Heap space in order to complete the report generation operation, it might be the case you executed a long-running test and your .jtl results file is very big so it doesn't fit into 7GB of heap. It normally indicates that a developer is not too familiar with memory management and cannot come up with a better solution than load everything into memory in one shot instead of doing it in batches/buffers so it might be a good idea to raise an issue in JMeter's Bugzilla
So just increase JVM heap size allocated to JMeter by manipulating -Xmx parameter, even if your system will start intensively using the swap file I believe you should be able to live with this.
Alternative option is generating tables/charts using JMeter Plugins Command Line Graph Plotting Tool, however this way you will get individual results rather than the fancy HTML dashboard
It starts w/the proverbial:
[Notes - F1 [107]] Error: An error occurred with the following error message: "System.OutOfMemoryException: Insufficient memory to continue the execution of the program. (SSIS Integration Toolkit for Microsoft Dynamics 365, v10.2.0.6982 - DtsDebugHost, v13.0.1601.5)".
But even in it's own diagnostics, it shows that plenty of memory is available (yes, that's 32GB I have on my system):
Error: The system reports 47 percent memory load. There are 34270687232 bytes of physical memory with 18094620672 bytes free. There are 4294836224 bytes of virtual memory with 981348352 bytes free. The paging file has 34270687232 bytes with 12832284672 bytes free.
The info messages report memory pressure:
Information: The buffer manager failed a memory allocation call for 506870912 bytes, but was unable to swap out any buffers to relieve memory pressure. 2 buffers were considered and 2 were locked. Either not enough memory is available to the pipeline because not enough are installed, other processes were using it, or too many buffers are locked.
I currently have the max rows set at 500 w/the buffer size at 506,870,912 in this example. I've tried the maximum buffer size, but that fails instantly, and the minimum buffer size still throws errors. I've fiddled w/various sizes, but it never gets anywhere close to processing the whole data set. The error I get when I set the DefaultBufferSize lower is:
[Notes - F1 [107]] Error: An error occurred with the following error message: "KingswaySoft.IntegrationToolkit.DynamicsCrm.CrmServiceException: CRM service call returned an error: Failed to allocate a managed memory buffer of 536870912 bytes. The amount of available memory may be low. (SSIS Integration Toolkit for Microsoft Dynamics 365, v10.2.0.6982 - DtsDebugHost,
I've looked for resources on how to tune this, but cannot find anything relevant to having a 64bit Window 10 machine (as opposed to a server) that has 32GB of RAM to play with.
For a bit more context, I'm migrating notes from one CRM D365 environment to another using Kingsway. The notes w/attachments are the ones causing the issue.
Properties:
Execution
Source
Destination
I have had this problem before and it was not the physical memory (i.e., RAM), but the physical disk space where the database is stored. Check to see what the available hard drive space is on the drive that stores both the database and transaction log files - chances are that it is full and therefore unable to allocate any additional disk space.
In this context, the error message citing 'memory' is a bit misleading.
UPDATE
I think this is actually caused by having too much data in the pipeline buffer. You will need to either need to look at expanding the buffer's memory allocation (i.e., DefaultBufferSize) or you will need to take a look at what data is flowing through the pipeline. Typical causes can be a lot of columns with large NVARCHAR() character counts. Copying the rows with MultiCast will only compound the problem. With respect to the 3rd party component you are using, your guess is as good as mine because I have not used them.
For anyone coming along later:
The error says "CRM service call returned an error: Failed to allocate a managed memory buffer of 536870912 bytes". I understood it to be the CRM Server that had the memory issue.
Regardless, we saw this error when migrating email attachments via the ActivityMimeAttachment entity. The problem appeared to be related to running the insert to the target CRM with too large a batch size and/or multi-threaded.
We set the batch size to 1 and turned off the multi-threading and the issue went away. (We also set the batch size to 1 on the request from the source - we saw "service unavailable" errors from an on-premise CRM when the batch size was too high and the attachments were too large.)
We got "device overload" error after the program ran successfully on production for a few months. And we find that some maps' sizes are very big, which may be bigger than 1,000.
After I inspected the source code, I found that the reason of "devcie overload" is that the write queue is beyond limitations, and the length of the write queue is related to the effiency of processing.
So I checked the "particle_map" file, and I suspect that the whole map will be rewritten even if we just want to insert one pair of KV into the map.
But I am not so sure about this. Any advice ?
So I checked the "particle_map" file, and I suspect that the whole map will be rewritten even if we just want to insert one pair of KV into the map.
You are correct. When using persistence, Aerospike does not update records in-place. Each update/insert is buffered into an in-memory write-block which, when full, is queued to be written to disk. This queue allows for short bursts that exceed your disks max IO but if the burst is sustained for too long the server will begin to fail the writes with the 'device overload' error you have mentioned. How far behind the disk is allowed to get is controlled by the max-write-cache namespace storage-engine parameter.
You can find more about our storage layer at https://www.aerospike.com/docs/architecture/index.html.
I'm performing some queries over a tpch 100gb dataset on presto, I have 4 nodes, 1 master, 3 workers. When I try to run some queries, not all of them, I see on Presto web interface that the nodes die during the execution, resulting in query failure, the error is the following:
.facebook.presto.operator.PageTransportTimeoutException: Encountered too many errors talking to a worker node. The node may have crashed or been under too much load. This is probably a transient issue, so please retry your query in a few minutes.
I rebooted all nodes and presto service but the error remains, this problem doesn't exist if I run the same queries over a smaller dataset.Can someone provide some help on this problem?
Thanks
3 possible causes for this kind of error. You may ssh into one of worker to find out what the problem is when the query is running.
High CPU
Tune down the task.concurrency to, for example, 8
High memory
In the jvm.config, -Xmx should no more than 80% total memory. In the config.properties, query.max-memory-per-node should be no more than the half of Xmx number.
Low open file limit
Set in the /etc/security/limits.conf a larger number for the Presto process. The default is definitely way too low.
It might be an issue for configuration. For example, if the local maximum memory is not set appropriately and the query use too much heap memory, full GC might happen to cause such errors. I would suggest to ask in the Presto Google Group and describe someway to reproduce the issue :)
I was running presto on Mac with 16GB of ram below is the configuration of java.config file.
-server
-Xmx16G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError=kill -9 %p
I was getting following error even for running the Query
Select now();
Query 20200817_134204_00005_ud7tk failed: Encountered too many errors talking to a worker node. The node may have crashed or be under too much load. This is probably a transient issue, so please retry your query in a few minutes.
I changed my -Xmx16G value to -Xmx10G and it works fine.
I used following link to install presto on my system.
Link for Presto Installation
I have CSV file of around 188MB. When I try to upload data using hot folder technique its taking too much time 10-12 hrs. How can I speedup the data upload?
Thanks
Default value of impex.import.workers is 1. Try to change this value. And I recommend making performance test with a bit smaller file first, than 188Mb (just to get swift results)
Adjust the number of impex threads on the backoffice server to speed up ImpEx file processing. It is recommended that you start with it equal to the number of cores available on a backoffice node. You should not adjust it any higher than 2 * number of cores, and this is only if the IMPEX processes will be the only item running on the node. The actual value could be somewhere in between and will only be determined by testing and analyzing the number of other processes, jobs, apps running on your server to ensure you are not maxing out CPU.
NOTE: this value could be higher for lower environments since Hybris will likely be the only process running.
Taken from Tuning Parameters - Hybris Wiki