I have a sample text file named testfile.txt containing simple "Hi". I want this data to get indexed on ElasticSearch. I run the following command on logstash:
bin/logstash -f logstash-test.conf
Conf File content is below:
input{
file
{
path=> "/home/abhinav/ELK/logstash/testfile.txt"
type => "test"
start_position => "beginning"
}
}
output{
elasticsearch { host => localhost
index => "test_index"
}
stdout { codec => rubydebug }
}
The ElasticSearch Log shows the follwing error:
[2015-05-04 14:52:23,082][INFO ][cluster.service ] [Argo]
added
{[logstash-abhinav-HP-15-Notebook-PC-10919-4006][CPk1djqFRnO-j-DlUMJIzg][abhinav-HP-15-Notebook-PC][inet[/192.168.1.2:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[logstash-abhinav-HP-15-Notebook-PC-10919-4006][CPk1djqFRnO-j-DlUMJIzg][abhinav-HP-15-Notebook-PC][inet[/192.168.1.2:9301]]{client=true,
data=false}])
I tried following things:
Tried with simple std input(stdin) to ES n stdout . It worked.
If you are using the same file repeatedly to test with, you are running into the "sincedb" problem -- see How to force Logstash to reparse a file?
You need to add sincedb_path => "/dev/null" to your file input. Generally this is not needed in a production scenario, but it is when testing.
I have found solution to my problem and also found things we can check, if logstash & ES not working/communicating properly. :
Make sure ES & Logstash are properly installed
Versions installed are compatible with each other
Even while testing, try making different Logstash conf file for different test case, as suggested by Mr. Alcanzar in above comment.
You can also refer to below links , for help regarding this issue :
Cannot load index to elasticsearch from external file, using logstash
https://logstash.jira.com/browse/LOGSTASH-1800
Related
New to logstash and following the tutorial posted https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html
Trying to set up my first-pipeline.conf where in i need to specify the input , filter and the output configurations
When i specify these configurations , and try
logsstash -f first-pipeline.conf -configtest i get a RuntimeError
RuntimeError : translation missing : en.logstash.runner.configuration.file-not-found>, class=> RuntimeError : backtrace => ["C:/ELK/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lob/logstash/config/loader/rb:58 in 'local_config" and bunch of other stack trace
Here below is the snip of stack trace
looks like im missing some files in my logstash installation direcotory..
BTW here is what my first-pipeline.conf file looks like
Also , i commented out the filter portion of my first-pipeline.conf as was not sure if grok was causing this issue and still the same error is reproducible
The error "io/console not supported; tty will not be manipulated" seems a jruby bug:
https://github.com/jruby/jruby/issues/3550
And It seems to be fixed on version 1.7.24. In Logstash 2.3.2 jruby version is 1.7.23 that the bug opened for. So you can try to download jruby 1.7.25 and replace it with the one under vendor/jruby.
For the other error, you are running Logstast from bin folder. Is your configuration (first-pipeline.conf) file actually in that folder? If not specify it from where it is.
I'm following instructions on
https://laravelcollective.com/docs/5.1/ssh
in order to use SSH to perform SFTP download from a private server.
I've done so far:
$> composer require laravelcollective/remote
added in config app :
'providers' => [
....
Collective\Remote\RemoteServiceProvider::class,
...
]
'aliases' => [
....
'SSH' => Collective\Remote\RemoteFacade::class,
...
]
published it:
$> php artisan vendor:publish --provider="Collective\Remote\RemoteServiceProvider"
Then I also run a composer update
But still in my console command if I test it like:
$contents = SSH::into('production')->getString('/hi.txt');
dd($contents);
I get the error in my question.
When a service provider is defined like above, the class is globally accessible? Or still I need to put the directive use Path/to/Class ?
If so, since the Alias ahas a different name from the real classname, how should I specify the use path directive?
use Collective\Remote\RemoteServiceProvider
or
use Collective\Remote\RemoteServiceProvider
?
What am I missing?... I've tested other preconfigured services that comes with laravel 5.2 fresh install (i.e. Redis) and they seems to be found without any additional use directive in class.....
Thanks
Just use it as global class \SSH and it will work.
$contents = \SSH::into('production')->getString('/hi.txt');
dd($contents);
I am new to Impress Pages. Using version 4.2.7 ( Installed 4.2.6 and upgraded from within the CMS). I have the following issues;
Tried to add the "PHP" plugin and got error message
"Plugin signature verification failed" . Plugin is NOT ADDED
I tried to add a new theme and got the error message Unknown error. Please see logs.
Where are these logs ? I have tried /var/log/apache2/error_log , etc.
Update 1: The log within the cms shows ;
2014-11-01 17:00:44 Cron.finished array(0) { } 2014-11-01 17:00:44
Cron.started array(7) { 'firstTimeThisYear' => bool(false)
'firstTimeThisMonth' => bool(false) 'firstTimeThisWeek' => bool(false)
'firstTimeThisDay' => bool(false) 'firstTimeThisHour' => bool(true)
'lastTime' => int(1414828943) 'test' => NULL }
I dont see anything related to plugins or themes ? Is there another log ?
The error says that the signature of the file can't be verified. There may be two reasons for that:
Your server is missing some encryption lib. But I guess current implementation should not require any additional libs.
The downloaded file is not what had to be downloaded. That could be due to DNS errors, network error, etc.
Which case it is can be told only by debugging. You can always download plugins from market.impresspages.org and install them manually.
If you want to debug, first place to look at is file/secure/tmp dir. You should find the downloaded plugin there. Have a look if you can unzip it. If you can un-zip, you have the problem number 1, otherwise 2.
Most likely your server couldn't download files from Internet.
You can
Does logstash use its own file syntax in config file? Is there any parser or validator for config file syntax?
For anyone that does not use logstash but have idea about file formats here is a sample syntax:
input {
file {
path => "/var/log/messages"
type => "syslog"
}
file {
path => "/var/log/apache/access.log"
type => "apache"
}
}
The Logstash configuration file is a custom format developed by the Logstash folks using Treetop.
The grammar itself is described in the source file grammar.treetop and compiled using Treetop into the custom grammar.rb parser.
That parser is then used by the pipeline.rb file in order to set up the pipeline from the Logstash configuration.
If you're not that much into Ruby, there's another interesting project called node-logstash which provides a Logstash implementation in Node.js. The configuration format is exactly the same as with the official Logstash, though the parser is obviously a different one written for Node.js. In this project, the Logstash configuration file grammar is described in jison and the parser is also automatically generated, but could be used by any Node.js module simply by requiring that generated parser.
You can use the following command to verify your logstash config file is valid:
bin/logstash --configtest --config <your config file>
Apparently command line arguments have been updated since the answers have been posted and --configtest and --config arguments are no longer valid. In order to ask Logstash (at least v5) to validate config file:
bin/logstash -t -f config.conf
With expanded arguments it looks like this:
bin/logstash --config.test_and_exit --path.config config.conf
So far. there is no any parser or validator for logstash config. You can only use the logstash to verify the config.
For more information about config, you can visit here. All the format is introduced in this page.
The Logstash-Config Go package config provides a ready to use parser and Abstract Syntax Tree for Logstash configuration files in Go.
The basis of the grammar for the parsing of the Logstash configuration format is the original Logstash Treetop grammar .
logstash-config uses pigeon to generate the parser from the PEG (parser expression grammar).
Every time when I try to start my mapreduce application (in standalone Hadoop), it tries to put stuff in the tmp directory, which it can't:
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-username\mapred\staging\username-1524148556\.staging to 0700
It ties to use an invalid path (slashes should be the other way around for cygwin).
I set hadoop.tmp.dir in core-site.xml (in the conf folder of Hadoop), but it seems that the config file is never read (if I put syntax errors in the file, it makes no difference). I added:
--config /home/username/hadoop-1.0.1/conf
To the command, but no difference. I also tried:
export HADOOP_CONF_DIR=/home/username/hadoop-1.0.1/conf
but also that does not seem to have an effect....
Any pointers on why the configs would not be read, or what else I am failing to see here?
Thanks!
It's not that the slashes are inverted, it's that /tmp is a cygwin path which actually maps to /cygwin/tmp or c:\cygwin\tmp. since hadoop is java and doesn't know about cygwin mappings, it takes /tmp to mean c:\tmp.
there's an awful lot of stuff to patch if you want to get 1.0.1 running on cygwin.
see: http://en.wikisource.org/wiki/User:Fkorning/Code/Hadoop-on-Cygwin
I found the following link useful, it seems that the problem stands with newer version of Hadoop. I'm using version 1.0.4 and I'm still facing this problem.
http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/25837
UPDATED: in Mahout 0.7 and for the ones who use the "Mahoot in Action" book example, you shoud change the example code as follows:
File outFile = new File("output");
if (!outFile.exists()) {
outFile.mkdir();
}
Path output = new Path("output");
HadoopUtil.delete(conf, output);
KMeansDriver.run(conf, new Path("testdata/points"), new Path("testdata/clusters"),
output, new EuclideanDistanceMeasure(), 0.001, 10,
true, 0.1, true);