Logstash|exception=>"LogStash::ConfigurationError",:message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 22, column 20 - google-bigquery
Recently I encountered a little white problem when I was learning logstash,I try to transfer data from Kafka to Bigquery
I'm instantiating logstash using -
logstash-8.2.3/bin/logstash -f config/logstash-sample.conf
Config File
input {
kafka {
bootstrap_servers => ["addcn102:9092,addcn103:9092,addcn104:9092"]
group_id => "logstash_kagka"
auto_offset_reset => "earliest"
consumer_threads => 3
topics => ["houstonbq"]
codec => "json"
}
}
output {
google_bigquery {
project_id => "newcar8891"
dataset => "logstash"
json_key_file => "/home/shurui/bin/newcar8891-013ef3777de7.json"
error_directory => "/opt/module/bqerror"
flush_interval_secs => "30"
csv_schema >= "message:STRING"
}
}
This is error log
Using bundled JDK: /opt/module/logstash-8.2.3/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /opt/module/logstash-8.2.3/logs which is now configured via log4j2.properties
[2022-10-21T17:19:07,041][INFO ][logstash.runner ] Log4j configuration path used is: /opt/module/logstash-8.2.3/config/log4j2.properties
[2022-10-21T17:19:07,051][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-10-21T17:19:07,052][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.2.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-10-21T17:19:07,054][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-10-21T17:19:07,349][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-10-21T17:19:08,125][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-10-21T17:19:08,401][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError",
:message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 19, column 13 (byte 420)
after output
{\n google_bigquery {\n\t
project_id => \"newcar8891\"\n\t
dataset => \"logstash\"\n\t
csv_schema ",:backtrace=>["/opt/module/logstash-8.2.3/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'",
"org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'",
"/opt/module/logstash-8.2.3/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'",
"/opt/module/logstash-8.2.3/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'",
"/opt/module/logstash-8.2.3/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}
[2022-10-21T17:19:08,471][INFO ][logstash.runner ] Logstash shut down.
This is my kafka data
{
"#timestamp" => 2022-10-21T09:24:51.281353Z,
"message" => "help me \b",
"tags" => [
[0] "_jsonparsefailure"
],
"#version" => "1"
}
I want to send the content of the message to bigquery
From error message it can be understood that there is a syntax error in your output plugin configuration. Try below output plugin in config.
output {
google_bigquery {
project_id => "newcar8891"
dataset => "logstash"
csv_schema => "message:STRING"
json_key_file => "/home/shurui/bin/newcar8891-013ef3777de7.json"
error_directory => "/opt/module/bqerror"
flush_interval_secs => "30"
}
}
Related
LogStash catch my logs only with debug mod
I have a little problem on logstash. When I test in debug mode everything is OK with the command
sudo /usr/share/logstash/bin/logstash --debug --path.settings /etc/logstash -f /etc/logstash/conf.d/apache.conf
But if I start logstash in service with systemctl (systemctl start logstash), it starts but does not trap any log. Nothing happens (the logstash logs do not move ...). All commands in sudo or root.
Logstash and Apach are not on the same machine as Elasticsearch (but it works since in debug they communicate)
Do you know why ?
EDIT
This is the pipeline configuration
input {
file {
path => "/var/log/httpd/access_log"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
convert => {
"response" => "integer"
"bytes" => "integer"
}
}
}
output {
elasticsearch {
hosts => "192.168.1.2:9200"
index => "apache-int-%{+YYYY.MM.dd}"
}
}
And extraction of logstash log id available here : https://sharetext.me/U9RAMT
Well, I just found ... When I started debug mode, I was root, when systemctl starts logstash, it uses user logstash which does not have access to httpd logs! With the right rights, it works better!
'lessc' is not recognized as an internal or external command, Yii2 WAMP win8
So i normally work on external servers, but decided to set up a local environment in the form of WAMPserver on my windows 8.1 computer. From (git bash) console, i can run lessc without a problem, but when i install Yii2, and attempt to process .less it says it can't find the command. AssetConverter command 'lessc "C:/wamp/www/basic/web/css/site.less" "C:/wamp/www/basic/web/css/site.css" --no-color --source-map' failed with exit code 1: STDOUT: STDERR: 'lessc' is not recognized as an internal or external command, operable program or batch file. Now as far as i can tell, this should work. In fact, when i run that same command from console it works just fine. Am i missing a step here? Any help would be appreciated.
I fixed it just now.
How should you do?
Make sure you can run lessc in cmd windows.If you can't,install it with node.js Install lessc with node.js
Make sure lessc path in system path.Mine is C:\Users\l7861\AppData\Roaming\npm or you can type where lessc in cmd windows to find it.
Make sure you can see lessc path in path_info().If you can't restart your apache server.
Finally you don't need to add addassetManager in config never~~~~~
This below is a discovery process.
I just thought it is a bug on yii2 at first.So I debug it with Phpstrom.And I saw this line:path\to\vendor\yiisoft\yii2\web\AssetConverter.php 91
$command = Yii::getAlias($command);
$command = strtr($command, [
'{from}' => escapeshellarg("$basePath/$asset"),
'{to}' => escapeshellarg("$basePath/$result"),
]);
$descriptor = [
1 => ['pipe', 'w'],
2 => ['pipe', 'w'],
];
$pipes = [];
$proc = proc_open($command, $descriptor, $pipes, $basePath);
$stdout = stream_get_contents($pipes[1]);
$stderr = stream_get_contents($pipes[2]);
foreach ($pipes as $pipe) {
fclose($pipe);
}
$status = proc_close($proc);
if ($status === 0) {
Yii::trace("Converted $asset into $result:\nSTDOUT:\n$stdout\nSTDERR:\n$stderr", __METHOD__);
} elseif (YII_DEBUG) {
throw new Exception("AssetConverter command '$command' failed with exit code $status:\nSTDOUT:\n$stdout\nSTDERR:\n$stderr");
} else {
Yii::error("AssetConverter command '$command' failed with exit code $status:\nSTDOUT:\n$stdout\nSTDERR:\n$stderr", __METHOD__);
}
return $status === 0
}
Only if $status is not 0,then you got a error message.
And that usually means $command has something wrong.
When I ran to here, I could see it is 'lessc "path/to/frontend/web/less/main.less" "path/to/frontend/web/less/main.css" --no-color';
And I ran it in cmd.Then css file was created by it.
I wrote a similar code below like that function
$command='lessc "path/to/frontend/web/less/main.less" "path/to/frontend/web/less/main.css" --no-color';
$basePath="path/to/frontend/web";
$descriptor = [
1 => ['pipe', 'w'],
2 => ['pipe', 'w'],
];
$pipes = [];
$process = proc_open($command, $descriptor, $pipes /* $cwd, $env */ );
print_r($pipes);
$stdout = stream_get_contents($pipes[1]);
echo mb_convert_encoding($stdout, 'UTF-8', 'UTF-8,GBK,GB2312,BIG5' );
$stderr = stream_get_contents($pipes[2]);
echo mb_convert_encoding($stderr, 'UTF-8', 'UTF-8,GBK,GB2312,BIG5' );
if (is_resource($process)) {
fwrite($pipes[1], '<?php print_r($_ENV); ?>');
fclose($pipes[1]);
echo stream_get_contents($pipes[2]);
fclose($pipes[2]);
$return_value = proc_close($process);
echo "command returned $return_value\n";
}
And ran it in browser.Then I got a error "lessc is not recognized as an internal or external command".
So I chang config file config/main.php,
return [
'vendorPath' => dirname(dirname(__DIR__)) . '/vendor',
'components' => [
'cache' => [
'class' => 'yii\caching\FileCache',
],
'assetManager' => [
'converter' => [
'class' => 'yii\web\AssetConverter',
'commands' => [
'less' => ['css', 'lessc {from} {to} --no-color'],
],
],
],
],
'name' => 'FavorTGD',
];
I got a same a same message.So I check out php_info(),there no lessc program in path.there there.That all you need is just restart your apache.
And delete assetManager in main.php
Ended up fixing it by changing the command to reference the lessc.cmd directly
'assetManager'=>[
'converter' => [
'class' => 'yii\web\AssetConverter',
'commands' => [
'less' => ['css', 'c:\path\to\lessc.cmd {from} {to} --no-color'],
],
],
],
Not to happy about the fix, because this means i will need to change my settings when i go 'live', but nothing else worked. I tried symlinking it into the yii directory, commands directory, etc.. but no good.
Laravel 5.0 Sweet Alert Class 'UxWeb\SweetAlert\SweetAlertServiceProvider' not found
please i have done these :
"require": {
"uxweb/sweet-alert": "~1.1",
}
'providers' => [
'UxWeb\SweetAlert\SweetAlertServiceProvider::class'
]
'aliases' => [
'Alert' => UxWeb\SweetAlert\SweetAlert::class
]
but i keep getting the error.
I was working under proxy therefore my internet connection was not working in the command line hence, composer update Was not working
Install apache module with puppet on Debian 7
My problem is during the provisionning of my Vagrant VM using puppet. The error is :
==> default: err: /Stage[main]//Service[apache2]: Could not evaluate: Could not find init script for 'apache2'
My puppetFile is the following default.pp :
Exec { path => [ "/bin/", "/sbin/" , "/usr/bin/", "/usr/sbin/" ] }
exec { 'apt-get update':
command => 'apt-get update',
timeout => 60,
tries => 3
}
class { 'apt': }
$sysPackages = ['python-software-properties', 'build-essential', 'apache2', 'apache2.2-common']
package { $sysPackages:
ensure => "installed",
require => Exec['apt-get update'],
}
service { apache2:
ensure => running,
enable => true,
require => Package['apache2', 'apache2.2-common'],
}
When not writting the apache2 service, there is no error thrown, but there is no /etc/init.d/apache2. I guess that's why the error is thrown. So how can I avoid this error, or more generaly how to cleanly install this module ?
Any help is welcome.
Error 'invalid parameter include_src' when provisionning RabbitMQ with Puppet on Vagrant
I am trying to install the RabbitMQ module on my virtual machine. I used :
git submodule add https://github.com/puppetlabs/puppetlabs-rabbitmq.git
and I installed every dependecy I could find the same way (which are, like listed in this answer : stdlib, epel, staging and erlang).
I reduced my manifest to the bare minimum :
Exec { path => [ "/bin/", "/sbin/" , "/usr/bin/", "/usr/sbin/" ] }
exec { 'apt-get update':
command => 'apt-get update',
timeout => 60,
tries => 3
}
package { ['python-software-properties']:
ensure => 'installed',
require => Exec['apt-get update'],
}
$sysPackages = [ 'build-essential', 'git', 'curl']
package { $sysPackages:
ensure => "installed",
require => Exec['apt-get update'],
}
include epel
include staging
class { 'erlang': epel_enable => true}
class { 'rabbitmq':
port => '5672',
service_manage => true,
environment_variables => {
'RABBITMQ_NODENAME' => 'server',
'RABBITMQ_SERVICENAME' => 'rabbitMQ'
}
}
And when I provision my VM with vagrant provision, I get the error :
==> default: Error: Invalid parameter include_src at /tmp/vagrant-puppet/modules
-33d06c2339c7ee7ab2bd92b2d11cf5d9/erlang/manifests/repo/apt.pp:39
I'm almost certain it is because of a dependency I do not have, but I can not find out which one. Any ideas ?
It seems that erlang module requires older version of apt module. Look at this patch. Recently, in apt resource, parameters include_src and include_deb were substituted by one include parameter. To solve the problem please install apt module in version 1.8.0.