Just like to ask whether is there any character limit when fluentbit send logs to splunk? How can I increase the character limit via the Splunk Web UI?
It depends on how the data is sent to Splunk. If it comes in via HEC then the limit is 1 million bytes (not characters) and cannot be changed. If the data comes in via port 9997 then it does not have the same limit, but will be truncated at 10,000 bytes. The truncation can be set for the sourcetype via the TRUNCATE attribute in props.conf or at Settings->Source types.
Create a new file called props.conf in "C:\Program Files\Splunk\etc\system\local"
For truncate setting in the props.conf local
[host::small_events]
TRUNCATE = 256
For more info:
https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/Propsconf
https://docs.splunk.com/Documentation/Splunk/9.0.2/Admin/Wheretofindtheconfigurationfiles#:~:text=Splunk%20software%20uses%20configuration%20files,the%20system%20as%20a%20whole.
Related
How can we reduce the file size for aerospike .dat file?
our current config is
namespace test {
memory-size 20G # Maximum memory allocation for data and
# primary and secondary indexes.
storage-engine device { # Configure the storage-engine to use
file /opt/aerospike/test.dat # Location of data file on server.
filesize 100G # Max size of each file in GiB.
}
}
current file size if test.dat is 90GB as par ls -ltrh. But on AMC ui it shows 50GB is used.
I want to reduce the file size to 80GB. I tried following this doc
Decrease
filesize Decreasing the size of the files with an Aerospike
service restart will potentially end up deleting random data which can
results in unexpected behavior on the Aerospike cluster due to the
truncation, maybe even landing into low available percentage on the
node. Thus, you would need to delete the file itself and let the data
be migrated from the other nodes in the cluster.
Stop Aerospike server.
Delete the file and update the configuration with the new filesize.
Start Aerospike server.
But when I start the server post data deletion, the startup fails with error
Jan 20 2022 03:44:50 GMT: WARNING (drv_ssd): (drv_ssd.c:3784) unable to open file /opt/aerospike/test.dat: No such file or directory
I have few questions wrt this
Is there a way to restart process with no initial data and let it take data from other nodes in the cluster?
If i wanted to reduce the size from 100G to 95G, would i still have to do the same thing? considering current file size is only 90GB. Is there still a risk of losing data?
Stopping the Aerospike server, deleting the file and restarting it is the way to go. There are other ways (like cold starting empty -- cold-start-empty) but the way you have done it is the recommended one. Seems there are some permission issues preventing the server to create that file in that directory.
Yes, you would have to do the same thing for reducing the file size, as mentioned in that document you referred to.
I have set of files on Azure Data-lake store folder location. Is there any simple power-shell command to get the count of records in a file? I would like to do this with out using Get-AzureRmDataLakeStoreItemContent command on the file item as the size of the files in gigabytes. Using this command on big files is giving the below error.
Error:
Get-AzureRmDataLakeStoreItemContent : The remaining data to preview is greater than 1048576 bytes. Please specify a
length or use the Force parameter to preview the entire file. The length of the file that would have been previewed:
749319688
Azure data lake operates at the file/folder level. The concept of a record really depends on how an application interprets it. For instance, in one case the file may have CSV line or in another a set of JSON objects. In some cases files contain binary data. Therefore, there is no way at the file system level to get the count of records.
The best way to get this information is to submit a job such as a USQL job in Azure Data Lake Analytics. The script will be really simple: An EXTRACT statement followed by a COUNT aggregation and an OUTPUT statement.
If you prefer Spark or Hadoop here is a StackOverflow question that discusses that: Finding total number of lines in hdfs distributed file using command line
First, sorry for my approximative english.
I'm a little lost with HSQLDB using.
I need to save in local database a big size of data (3Go+), in a minimum of time.
So I made the following :
CREATE CACHED TABLE ...; for save data in .data file
SET FILES LOG FALSE; for don't save data in .log file and gain time
SHUTDOWN COMPACT; for save records in local disk
I know there's other variable to parameter for increase the .data size and for increase data access speed, like :
hsqldb.cache_scale=
hsqldb.cache_size_scale=
SET FILES NIO SIZE xxxx
But I don't know how to parameter this for a big storage.
Thanks to help me.
When you use SET FILES LOG FALSE data changes are not saved until you execute SHUTDOWN
or CHECKPOINT.
The other parameters can be left to their default values. If you want to use more memory and gain some speed, you can multiply the default values of the parameters by 2 or 4.
I am trying to load data into AWS redshift using following command
copy venue from 's3://mybucket/venue'
credentials 'aws_access_key_id=<access-key-id>;aws_secret_access_key=<secret-access-key>'
delimiter '\t';
but data load is failing, when I checked Query section for that specific load I noticed it failed because of "Bad UTF8 hex sequence: a4 (error 3)"
Is there a way to skip bad records in data load into redshift?
Yes, you can use the maxerror parameter. This example will allow up to 250 bad records to be skipped (the errors are written to stl_load_errors):
copy venue
from 's3://mybucket/venue'
credentials 'aws_access_key_id=;aws_secret_access_key='
delimiter '\t'
maxerror as 250;
When I open my table in phpMyAdmin and click Export (selecting .zip from the save-as options), I get an empty (0 byte) zip file. The table has ~200 000 records. This problem does not occur for smaller tables. This problem also does not occur when I export only the first 60 000 records.
Is this a bandwidth limitation? Is this something in my settings for MySQL?
This is potentially not a MySQL but a PHP problem, you might be exceeding the memory limit for a PHP script, try increasing memory_limit in the php.ini.
Also, you can directly exporting by using cmd in the dir of that database instead of exporting using the phpMyAdmin UI.
C:>mysql -u username -p password database_name < filename.sql