Amazon CloudWatch log agent ignoring first character of log lines - amazon-cloudwatch

If I add a batch of n test records to my log file, the awslogs agent is erroneously ignoring the first character of the first line of the batch of rows that I add. This is fully reproducible. So if I use vi to append the following test log lines:
2017-06-16 15:20:01,123 this line will not get correctly parsed. first character in date string will be skipped.
2017-06-16 15:21:23,456 this will get parsed. 1
2017-06-16 15:22:23,456 this will get parsed. 2
2017-06-16 15:23:23,456 this will get parsed. 3
2017-06-16 15:24:23,456 this will get parsed. 4
2017-06-16 15:25:23,456 this will get parsed. 5
2017-06-16 15:26:23,456 this will get parsed. 6
2017-06-16 15:27:23,456 this will get parsed. 7
The leading 2 in the first row gets omitted by the log agent. In the CloudWatch Logs web console, the event shows up as 017-06-16 15:20:01,123 this line will..., the datetime string does not get successfully parsed, and the log event must use the timestamp of the previous log.
In the common scenario where I add log events to the file one at a time, the first letter of each line is ignored and the timestamp strings do not get correctly parsed. If I append multiple lines in vi before hitting :w save, only the first line experiences this problem and the other lines in the batch get ingested correctly.
I created the log file (as a test) with touch and have only added lines manually with vi so I don't think this is a file encoding problem.
I'm using a mostly standard default configuration.
My CloudWatch Agent Config File:
[general]
state_file = /var/awslogs/state/agent-state
[/var/log/myapp/app.log]
file = /var/log/myapp/app.log
log_group_name = MyAppLogGroup
log_stream_name = MyAppLogStream
datetime_format=%Y-%m-%d %H:%M:%S,%f
Then I download the latest setup script from https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
And run sudo ./awslogs-agent-setup.py -n -r us-west-2 -c cloudwatch_logs.config

Try setting the
initial_position = start_of_file
option in your config file explicitly, do you get the same behavior?

Related

New Windows installation, CLI not picking up .bigqueryrc

I had a Windows 10 machine working great with bq/gloud and gsutils all working great. Had to move to new machine and setting all of this back up. I cannot seem to get bq to recognize the entries in my .bigqueryrc file which was copied from my working machine. On my working machine, I am not using BIGQUERYRC environmental file, but just HOME, ie ~/.bigqueryrc
I say it does not seem to recognize is because if I introduce a random character I get an error about my .bigqueryrc file.
Let me first show that bq seems to be operating, overall, as expected
C:\Users\boyer>bq ls --project_id=broad-tapestry-sbx-boyer-14
datasetId
---------------
Broad_DataGov
Broad_EDW
Broad_Kitchen
Broad_Lake
Broad_Tableau
Broad_Utils
SHELL> bq ls brings nothing. The contents of my .biqqueryrc file is as thus:
credential_file = C:\Users\boyer\AppData\Roaming\gcloud\legacy_credentials\boyer#broadinstitute.org\singlestore_bq.json
project_id = broad-tapestry-sbx-boyer-14
[query]
--use_legacy_sql=false
Here is showing where my .bigqueryrc file is:
:\Users\boyer>ls -ls ~/.bigqueryrc
1 -rwx------+ 1 CHARLES+boyer CHARLES+boyer 198 Dec 22 12:48 /cygdrive/c/Users/boyer/.bigqueryrc
So, I try this and nothing happens either:
C:\Users\boyer>bq ls --bigqueryrc=c:/users/boyer/.bigqueryrc
C:\Users\boyer>bq ls --bigqueryrc=c:\users\boyer\.bigqueryrc
C:\Users\boyer>
But here is the error I purposefully introduced to see if it is actually looking at the file and it seems it is finding it.
C:\Users\boyer>bq ls --bigqueryrc=c:\users\boyer\.bigqueryrc
Unknown flag x found in bigqueryrc file in section global
The error above was introduced by adding a random 'x' character to the top of the file:
x
credential_file = C:\Users\boyer\AppData\Roaming\gcloud\legacy_credentials\boyer#broadinstitute.org\singlestore_bq.json
project_id = broad-tapestry-sbx-boyer-14
[query]
--use_legacy_sql=false

Jmeter non GUI mode csv report not showing latency

I am trying to get the jmeter html report for file transfer in SFTP protocol.
I am using SSH SFTP Protocol plugin and added Simple Data Writer to that thread group.
I have created my own sftp server using Apache MINA. Jmeter script will hit the server which i created and uploads the file.
Script Parameters:
Thread Group - 250
Ramp up period - 50
Loop Count - 1
After running the script in non GUI mode as nohup sh jmeter.sh -n -t Singlepart_MultipleThread_RampUp.jmx -l Singlepart_MultipleThread_RampUp.jtl. I do get a csv generated which i convert into html report command jmeter -g <csv> -o <destination_folder>.
The html report created shows Latency vs Time and Latency vs Request as zero and even the csv report shows latency column as zero.
Below is my user.properties file
user.properties
# Latencies Over Time graph definition
jmeter.reportgenerator.graph.latenciesOverTime.classname=org.apache.jmeter.report.processor.graph.impl.LatencyOverTimeGraphConsumer
jmeter.reportgenerator.graph.latenciesOverTime.title=Latencies Over Time
jmeter.reportgenerator.graph.latenciesOverTime.property.set_granularity=${jmeter.reportgenerator.overall_granularity}
# Latencies Vs Request graph definition
jmeter.reportgenerator.graph.latencyVsRequest.classname=org.apache.jmeter.report.processor.graph.impl.LatencyVSRequestGraphConsumer
jmeter.reportgenerator.graph.latencyVsRequest.title=Latencies Vs Request
jmeter.reportgenerator.graph.latencyVsRequest.exclude_controllers=true
jmeter.reportgenerator.graph.latencyVsRequest.property.set_granularity=${jmeter.reportgenerator.overall_granularity}
jmeter.properties
#---------------------------------------------------------------------------
# Results file configuration
#---------------------------------------------------------------------------
# This section helps determine how result data will be saved.
# The commented out values are the defaults.
# legitimate values: xml, csv, db. Only xml and csv are currently supported.
jmeter.save.saveservice.output_format=csv
# The below properties are true when field should be saved; false otherwise
#
# assertion_results_failure_message only affects CSV output
jmeter.save.saveservice.assertion_results_failure_message=true
#
# legitimate values: none, first, all
jmeter.save.saveservice.assertion_results=all
#
jmeter.save.saveservice.data_type=true
jmeter.save.saveservice.label=true
jmeter.save.saveservice.response_code=true
# response_data is not currently supported for CSV output
jmeter.save.saveservice.response_data=true
# Save ResponseData for failed samples
jmeter.save.saveservice.response_data.on_error=false
jmeter.save.saveservice.response_message=true
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.thread_name=true
jmeter.save.saveservice.time=true
jmeter.save.saveservice.subresults=true
jmeter.save.saveservice.assertions=true
jmeter.save.saveservice.latency=true
# Only available with HttpClient4
#jmeter.save.saveservice.connect_time=true
jmeter.save.saveservice.samplerData=true
#jmeter.save.saveservice.responseHeaders=false
#jmeter.save.saveservice.requestHeaders=false
#jmeter.save.saveservice.encoding=false
jmeter.save.saveservice.bytes=true
# Only available with HttpClient4
jmeter.save.saveservice.sent_bytes=true
jmeter.save.saveservice.url=true
jmeter.save.saveservice.filename=false
jmeter.save.saveservice.hostname=false
jmeter.save.saveservice.thread_counts=true
jmeter.save.saveservice.sample_count=false
jmeter.save.saveservice.idle_time=true
# Timestamp format - this only affects CSV output files
# legitimate values: none, ms, or a format suitable for SimpleDateFormat
#jmeter.save.saveservice.timestamp_format=ms
#jmeter.save.saveservice.timestamp_format=yyyy/MM/dd HH:mm:ss.SSS
# For use with Comma-separated value (CSV) files or other formats
# where the fields' values are separated by specified delimiters.
# Default:
#jmeter.save.saveservice.default_delimiter=,
# For TAB, one can use:
#jmeter.save.saveservice.default_delimiter=\t
# Only applies to CSV format files:
# Print field names as first line in CSV
#jmeter.save.saveservice.print_field_names=true
# Optional list of JMeter variable names whose values are to be saved in the result data files.
# Use commas to separate the names. For example:
#sample_variables=SESSION_ID,REFERENCE
# N.B. The current implementation saves the values in XML as attributes,
# so the names must be valid XML names.
# By default JMeter sends the variable to all servers
# to ensure that the correct data is available at the client.
# Optional xml processing instruction for line 2 of the file:
# Example:
#jmeter.save.saveservice.xml_pi=<?xml-stylesheet type="text/xsl" href="../extras/jmeter-results-detail-report.xsl"?>
# Default value:
#jmeter.save.saveservice.xml_pi=
# Prefix used to identify filenames that are relative to the current base
#jmeter.save.saveservice.base_prefix=~/
# AutoFlush on each line written in XML or CSV output
# Setting this to true will result in less test results data loss in case of Crash
# but with impact on performances, particularly for intensive tests (low or no pauses)
# Since JMeter 2.10, this is false by default
#jmeter.save.saveservice.autoflush=false
So basically facing issue at two places:
How to get the latency value?
When i provide Ramp up value as 1, the script with Thread Group =50 takes around 16 seconds to complete the upload, whereas if i give Ramp up something other than 1 such as 10 then the script ends after 10 secs exact, irrespective of file is getting uploaded or not and providing vague results in html report as well.
Any idea how to solve this. Or need to do anything else in script.
You cannot as the plugin you're using doesn't call SampleResult.setLatency() function anywhere
theoretically it should be possible to request the functionality from the plugin developers
Setting 10 seconds ramp-up period for 50 virtual users means that JMeter starts with 1 virtual user and gradually increases the load to 50 within 10 seconds duration. Make sure to have enough loops defined in the Thread Group as you may run into the situation when 1st user has already finished uploading the file and was terminated and 2nd hasn't need started so you have maximum 1 user concurrency (it can be checked using Active Threads Over Time listener). See JMeter Test Results: Why the Actual Users Number is Lower than Expected for more detailed explanation if needed.

Uploading job fails on the same file that was uploaded successfully before

I'm running regular uploading job to upload csv into BigQuery. The job runs every hour. According to recent fail log, it says:
Error: [REASON] invalid [MESSAGE] Invalid argument: service.geotab.com [LOCATION] File: 0 / Offset:268436098 / Line:218637 / Field:2
Error: [REASON] invalid [MESSAGE] Too many errors encountered. Limit is: 0. [LOCATION]
I went to line 218638 (the original csv has a headline, so I assume 218638 should be the actual failed line, let me know if I'm wrong) but it seems all right. I checked according table in BigQuery, it has that line too, which means I actually successfully uploaded this line before.
Then why does it causes failure recently?
project id: red-road-574
Job ID: Job_Upload-7EDCB180-2A2E-492B-9143-BEFFB36E5BB5
This indicates that there was a problem with the data in your file, where it didn't match the schema.
The error message says it occurred at File: 0 / Offset:268436098 / Line:218637 / Field:2. This means the first file (it looks like you just had one), and then the chunk of the file starting at 268436098 bytes from the beginning of the file, then the 218637th line from that file offset.
The reason for the offset portion is that bigquery processes large files in parallel in multiple workers. Each file worker starts at an offset from the beginning of the file. The offset that we include is the offset that the worker started from.
From the rest of the error message, it looks like the string service.geotab.com showed up in the second field, but the second field was a number, and service.geotab.com isn't a valid number. Perhaps there was a stray newline?
You can see what the lines looked like around the error by doing:
cat <yourfile> | tail -c +268436098 | tail -n +218636 | head -3
This will print out three lines... the one before the error (since I used -n +218636 instead of +218637), the one that had the error, and the next line as well.
Note that if this is just one line in the file that has a problem, you may be able to work around the issue by specifying maxBadRecords.

How do I export into pdf in phantomjs with this link.

I have read about phantomjs and rasterizejs as well. But my link is this:
http://localhost:5601/#/dashboard/External?_g=(time:(from:'2014-12-31T16:00:00.000Z',mode:absolute,to:'2015-01-01T16:00:00.000Z'))&_a=(filters:!(),panels:!((col:10,id:'Count-of-Source-IPs-(External)',row:1,size_x:3,size_y:3,type:visualization),(col:4,id:'Protocols-(External)',row:4,size_x:3,size_y:2,type:visualization),(col:7,id:'Top-5-Source-IPs-with-Protocols-and-Source-Port-(External)',row:4,size_x:6,size_y:6,type:visualization),(col:1,id:'Top-5-Source-IPs-(External)',row:4,size_x:3,size_y:2,type:visualization),(col:1,id:'Top-5-Countries-with-Protocols-(External)',row:1,size_x:6,size_y:3,type:visualization),(col:1,id:'Geographical-of-External-(Source)',row:6,size_x:6,size_y:4,type:visualization),(col:7,id:'Action-(External)',row:1,size_x:3,size_y:3,type:visualization)),query:(query_string:(analyze_wildcard:!t,query:'*')),title:External)
How do I make it such that it works with this command:
phantom.js rasterize.js "http://localhost:5601/#/dashboard/External?_g=(time:(from:'2014-12-31T16:00:00.000Z',mode:absolute,to:'2015-01-01T16:00:00.000Z'))&_a=(filters:!(),panels:!((col:10,id:'Count-of-Source-IPs-(External)',row:1,size_x:3,size_y:3,type:visualization),(col:4,id:'Protocols-(External)',row:4,size_x:3,size_y:2,type:visualization),(col:7,id:'Top-5-Source-IPs-with-Protocols-and-Source-Port-(External)',row:4,size_x:6,size_y:6,type:visualization),(col:1,id:'Top-5-Source-IPs-(External)',row:4,size_x:3,size_y:2,type:visualization),(col:1,id:'Top-5-Countries-with-Protocols-(External)',row:1,size_x:6,size_y:3,type:visualization),(col:1,id:'Geographical-of-External-(Source)',row:6,size_x:6,size_y:4,type:visualization),(col:7,id:'Action-(External)',row:1,size_x:3,size_y:3,type:visualization)),query:(query_string:(analyze_wildcard:!t,query:'*')),title:External)" external.pdf
I have been getting syntax error because of that.
The problem is probably that the command is too long for your terminal and some of it is cut off.
You can either directly put it into the script or read it from stdin. For that you need to edit rasterize.js.
First you need to reduce the x in all system.args[x] where x is above 1 by 1. If you've done that, then you can call the script as
phantom.js rasterize.js external.pdf
or
cat file.url | phantom.js rasterize.js external.pdf
in the second case.
Put URL into script
Change
address = system.args[1];
to
address = "http://localhost....";
Read from pipe
You can put your long URL into some file and pass that file to stdin of the PhantomJS script.
Change
address = system.args[1];
to
address = system.stdin.read();

Maximum length of Command parameters in Windows 8 scheduler

I have a console application that takes input command line arguments. I have to schedule this task on Windows 8 using Win32.TaskScheduler. The problem is whenever my command line arguments length exceeds 450 characters, i get a warning like this :
Task registered task "TaskName" , but not all specified triggers will start the task. User Action: Ensure all the task triggers are valid as configured. Additional Data: Error Value: 2147944183.
And the task does not run eventually at the run time.
Is there any limit on command line arguments' length?
I encountered a similar issue with scheduled tasks and command line arguments that exceeded 260 characters. We got the same useless error that you did.
Eventually we had to change the command line arguments so they were smaller (length of file paths in argument list)
We had this issue on Windows Server 2012 R2 64 bit, I'm not sure if the bitness has anything to do with it...