Related
Recently I encountered a little white problem when I was learning logstash,I try to transfer data from Kafka to Bigquery
I'm instantiating logstash using -
logstash-8.2.3/bin/logstash -f config/logstash-sample.conf
Config File
input {
kafka {
bootstrap_servers => ["addcn102:9092,addcn103:9092,addcn104:9092"]
group_id => "logstash_kagka"
auto_offset_reset => "earliest"
consumer_threads => 3
topics => ["houstonbq"]
codec => "json"
}
}
output {
google_bigquery {
project_id => "newcar8891"
dataset => "logstash"
json_key_file => "/home/shurui/bin/newcar8891-013ef3777de7.json"
error_directory => "/opt/module/bqerror"
flush_interval_secs => "30"
csv_schema >= "message:STRING"
}
}
This is error log
Using bundled JDK: /opt/module/logstash-8.2.3/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /opt/module/logstash-8.2.3/logs which is now configured via log4j2.properties
[2022-10-21T17:19:07,041][INFO ][logstash.runner ] Log4j configuration path used is: /opt/module/logstash-8.2.3/config/log4j2.properties
[2022-10-21T17:19:07,051][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-10-21T17:19:07,052][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.2.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-10-21T17:19:07,054][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-10-21T17:19:07,349][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-10-21T17:19:08,125][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-10-21T17:19:08,401][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError",
:message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 19, column 13 (byte 420)
after output
{\n google_bigquery {\n\t
project_id => \"newcar8891\"\n\t
dataset => \"logstash\"\n\t
csv_schema ",:backtrace=>["/opt/module/logstash-8.2.3/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'",
"org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'",
"/opt/module/logstash-8.2.3/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'",
"/opt/module/logstash-8.2.3/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'",
"/opt/module/logstash-8.2.3/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}
[2022-10-21T17:19:08,471][INFO ][logstash.runner ] Logstash shut down.
This is my kafka data
{
"#timestamp" => 2022-10-21T09:24:51.281353Z,
"message" => "help me \b",
"tags" => [
[0] "_jsonparsefailure"
],
"#version" => "1"
}
I want to send the content of the message to bigquery
From error message it can be understood that there is a syntax error in your output plugin configuration. Try below output plugin in config.
output {
google_bigquery {
project_id => "newcar8891"
dataset => "logstash"
csv_schema => "message:STRING"
json_key_file => "/home/shurui/bin/newcar8891-013ef3777de7.json"
error_directory => "/opt/module/bqerror"
flush_interval_secs => "30"
}
}
I have created a SocialEngine module named MyAwesomeModule that adds a signup step to existing SocialEngine signup process. It is in application/modules/Myawesomemodule. Besides the files in this directory, it also depends on other files e.g. several created specifically for this module and placed in application/modules/User. The problem that I have is that when I build a package for this module in http://example.com/install/sdk/build, it only includes the files residing within the module directory itself. Other files that are required for this module to work and need to be copied to external directories (application/modules/User) are not included in the package.
I tried adding the list of external files inside package info file i.e.application/packages/module-myawesomemodule-4.0.0.json before building the package as follows:
"application\/modules\/User": {
"type": "directory",
"path": "application\/modules\/User",
"structure": [
{
"path": "Plugin\/Signup\/Phone.php",
"dir": false,
"file": true,
"perms": "0644",
"size": 0,
"sha1": null
},
{
"path": "Form\/Signup\/Phone.php",
"dir": false,
"file": true,
"perms": "0644",
"size": 0,
"sha1": null
},
{
"path": "controllers\/PhoneController.php",
"dir": false,
"file": true,
"perms": "0644",
"size": 0,
"sha1": null
}
]
}
But, the external files named in the above snippet are not copied to the package when building. Also, it seems that the external files I named in the JSON file are gone in the destination package after installation.
How can I solve this problem?
This is much easier than I thought, thanks to this SocialEngine Community post.
All is required to add every file you want to your package is to include it in your manifest.php file for the module. As such:
return array(
'package' =>
array(
'type' => 'module',
'name' => 'advancedsmsplugin',
'version' => '4.0.0',
'sku' => 'com.company.sms',
'path' => 'application/modules/Advancedsmsplugin',
'title' => 'Advanced SMS Plugin',
'description' => 'Advanced SMS Plugin',
'author' => 'Company Ltd.',
'callback' =>
array(
'class' => 'Engine_Package_Installer_Module',
),
'actions' =>
array(
0 => 'install',
1 => 'upgrade',
2 => 'refresh',
3 => 'enable',
4 => 'disable',
),
'directories' =>
array(
0 => 'application/modules/Advancedsmsplugin',
),
'files' =>
array(
0 => 'application/languages/en/advancedsmsplugin.csv',
1 => 'application/modules/User/Form/Signup/Phone.php',
2 => 'application/modules/User/Plugin/Signup/Phone.php',
3 => 'application/testpackage.html'
),
),
);
?>
I have installed logstash on 2 nodes to send the logs to RabbitMQ. SSL is configured on RabbitMQ listening 5671 port. I have configured both the logstash to push the logs to rabbitmq server on the 5671 port.
This is my configuration.
input {
file {
path => "/var/log/messages"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:system_auth_timestamp} %{SYSLOGHOST:system_auth_hostname} %{GREEDYDATA:command_issued}: %{GREEDYDATA:message}" }
add_tag => "syslog"
}
}
output {
rabbitmq {
exchange => "elasticsearch-exchange"
exchange_type => "direct"
key => "logstash-routing_key"
ssl => true
#verify_ssl => true
ssl_certificate_password => 'Password'
ssl_certificate_path => 'certfile'
ssl_version => "TLSv1.2"
host => "10.2.0.0"
vhost => "es_vhost"
durable => true
persistent => true
port => 5671
user => "admin"
password => "password"
heartbeat => "5"
}
stdout {
codec => rubydebug
}
}
This is the error I am getting in the logstash log.
{:timestamp=>"2017-12-26T07:22:32.708000+0000", :message=>"Pipeline aborted due to error", :exception=>java.util.concurrent.TimeoutException, :backtrace=>["com.rabbitmq.utility.BlockingCell.get(com/rabbitmq/utility/BlockingCell.java:77)", "com.rabbitmq.utility.BlockingCell.uninterruptibleGet(com/rabbitmq/utility/BlockingCell.java:111)", "com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(com/rabbitmq/utility/BlockingValueOrException.java:37)", "com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(com/rabbitmq/client/impl/AMQChannel.java:367)", "com.rabbitmq.client.impl.AMQConnection.start(com/rabbitmq/client/impl/AMQConnection.java:293)", "com.rabbitmq.client.ConnectionFactory.newConnection(com/rabbitmq/client/ConnectionFactory.java:648)", "com.rabbitmq.client.ConnectionFactory.newConnection(com/rabbitmq/client/ConnectionFactory.java:678)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.new_connection_impl(/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare/session.rb:505)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:281)", "RUBY.converting_rjc_exceptions_to_ruby(/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare/session.rb:467)", "RUBY.new_connection_impl(/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare/session.rb:500)", "RUBY.initialize(/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare/session.rb:136)", "RUBY.connect(/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare/session.rb:109)", "RUBY.connect(/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.15.0-java/lib/march_hare.rb:20)", "RUBY.connect(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-mixin-rabbitmq_connection-4.1.1-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:174)", "RUBY.connect!(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-mixin-rabbitmq_connection-4.1.1-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:131)", "RUBY.register(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-rabbitmq-3.1.0-java/lib/logstash/outputs/rabbitmq.rb:40)", "RUBY.register(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:75)", "RUBY.start_workers(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:181)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.start_workers(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:181)", "RUBY.run(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136)", "RUBY.start_pipeline(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:error}
{:timestamp=>"2017-12-26T07:22:35.710000+0000", :message=>"stopping pipeline", :id=>"main"}
This is the error I am getting in RabbitMQ logs.
=INFO REPORT==== 27-Dec-2017::05:44:27 ===
accepting AMQP connection <0.1228.0> (10.2.0.0:42187 -> 10.24.168.17:5601)
=WARNING REPORT==== 27-Dec-2017::05:44:35 ===
closing AMQP connection <0.1228.0> (10.2.0.0:42187 -> 10.24.168.17:5601):
client unexpectedly closed TCP connection
This is RabbitMQ conf
% This file managed by Puppet
% Template Path: rabbitmq/templates/rabbitmq.config
[
{rabbit, [
{cluster_nodes, {[rabbit#node01, rabbitmq#node02, rabbit#node03], disc}},
{cluster_partition_handling, ignore},
{tcp_listen_options,
[binary,
{packet, raw},
{reuseaddr, true},
{backlog, 128},
{nodelay, true},
{exit_on_close, false}]
},
{default_user, <<"admin">>},
{default_pass, <<"passowrd">>},
{handshake_timeout, 60000},
{tcp_listeners, []},
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/etc/rabbitmq/ssl_cert/testca/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl_cert/server/cert.pem"},
{keyfile,"/etc/rabbitmq/ssl_cert/server/key.pem"},
{password, "Password"},
{verify,verify_peer},
{versions, ['tlsv1.2']},
{fail_if_no_peer_cert,false}]},
{ssl_handshake_timeout, 5000}
{log_levels, [{autocluster, debug}, {connection, info}]}
]},
{kernel, [
]},
{rabbitmq_management, [
{listener, [
{port, 15672}
]}
]}
].
% EOF
I have even changed the SSL listener port to 5601 and tried just to make sure that this is not port conflict. I am hitting the wall everytime here.
There was a mismatch in hostname. I have resolved it by proving an FQDN in the /etc/hosts file. SSL is working fine now.
I walk around yii2 manual by copy this code in manual
'controllerMap' => [
'account' => 'yii\app\controllers\UserController',
'article' => [
'class' => 'yii\app\controllers\PostController',
'enableCsrfValidation' => false,
],
],
paste in netbean
just access it
already have UserController
Your namespace is wrong! If you look at documentation, the correct code is like this:
[
'controllerMap' => [
'account' => 'app\controllers\UserController',
'article' => [
'class' => 'app\controllers\PostController',
'enableCsrfValidation' => false,
],
],
]
But you have an extra yii in the beginning of the namespace, so yii\app\controllers\UserController must be app\controllers\UserController.
Also i think you have some problems in url rewriting. If I'm true, you can find an instrument for fixing it in http://www.yiiframework.com/doc-2.0/guide-runtime-routing.html#using-pretty-urls and yii2 url rewrite config.
I am trying to setup a rabbitmq cluster with puppet. I have nodeA that initiates the cluster and all other nodes should join nodeA's cluster using nodeA's hostname.
All other nodes execute the following code.
class { 'rabbitmq':
service_manage => false,
port => '5672',
delete_guest_user => true,
erlang_cookie => $erlang_hash,
config_cluster => true,
cluster_nodes => ['nodeA'],
cluster_node_type => 'ram',
wipe_db_on_cookie_change => true,
}
Nodes executing this puppet code can ping nodeA and puppet generates the following config file,
% This file managed by Puppet
% Template Path: rabbitmq/templates/rabbitmq.config
[
{rabbit, [
{cluster_nodes, {['rabbit#nodeA'], ram}},
{cluster_partition_handling, ignore},
{default_user, <<"guest">>},
{default_pass, <<"guest">>}
]}
].
% EOF
nodeA is a disc node and the node trying to join nodeA's cluster is a ram node.
Unfortunately a node executing my puppet code doesn't join nodeA's cluster.