Folks,
Need to convert the following request header to a different format:
RequestHeader set Date "%{TIME_WDAY}e"
The %t variable looks like :
t=1367272677754275
Would like the Date= to look like:
Date: Tue, 27 Mar 2007 19:44:46 +0000
How is this done?
Thanks!
You cannot do that with the documented functionality of mod_headers. This module only supports the follwing variables (from the doc):
%t The time the request was received in Universal Coordinated Time since the epoch (Jan. 1, 1970) measured in microseconds. The value is preceded by t=.
%D The time from when the request was received to the time the headers are sent on the wire. This is a measure of the duration of the request. The value is preceded by D=. The value is measured in microseconds.
%{FOOBAR}e The contents of the environment variable FOOBAR.
%{FOOBAR}s The contents of the SSL environment variable FOOBAR, if mod_ssl is enabled.
Unless you continually want to set an environment variable to your current date and pull it in using mod_env, I suggest you use mod_rewrite.
Correct answer here is a mod_headers.c patch to add additional authentication information required by AWS and GCS
Related
Hello I am using S3KeySensor to look for parquet files created in specific partition. As the file names are generated from spark like (part-00499-e91c1af8-4352-4de9*), what should be the bucket_key ? Below code is failing.
ex:bucket_key=f"inbound/phix/empnf/datasetdate={var_ds_date}/*.parquet"
s3_data_filechk=S3KeySensor(
task_id='s3_data_filechk',
bucket_name=data_bucket_name,
bucket_key=f"inbound/phix/empnf/datasetdate={var_ds_date}/*.parquet/",
timeout=60 * 30, # timeout in 30 minutes
poke_interval=60 * 5 # (seconds); checking file in every five minutes
)
The S3KeySensor has parameter wildcard_match: whether the bucket_key should be interpreted as a Unix wildcard pattern. You can use it to set the logic you wish.
Example:
sensor=S3KeySensor(
...
bucket_key="*.parquet",
wildcard_match=True
)
I am trying to get the jmeter html report for file transfer in SFTP protocol.
I am using SSH SFTP Protocol plugin and added Simple Data Writer to that thread group.
I have created my own sftp server using Apache MINA. Jmeter script will hit the server which i created and uploads the file.
Script Parameters:
Thread Group - 250
Ramp up period - 50
Loop Count - 1
After running the script in non GUI mode as nohup sh jmeter.sh -n -t Singlepart_MultipleThread_RampUp.jmx -l Singlepart_MultipleThread_RampUp.jtl. I do get a csv generated which i convert into html report command jmeter -g <csv> -o <destination_folder>.
The html report created shows Latency vs Time and Latency vs Request as zero and even the csv report shows latency column as zero.
Below is my user.properties file
user.properties
# Latencies Over Time graph definition
jmeter.reportgenerator.graph.latenciesOverTime.classname=org.apache.jmeter.report.processor.graph.impl.LatencyOverTimeGraphConsumer
jmeter.reportgenerator.graph.latenciesOverTime.title=Latencies Over Time
jmeter.reportgenerator.graph.latenciesOverTime.property.set_granularity=${jmeter.reportgenerator.overall_granularity}
# Latencies Vs Request graph definition
jmeter.reportgenerator.graph.latencyVsRequest.classname=org.apache.jmeter.report.processor.graph.impl.LatencyVSRequestGraphConsumer
jmeter.reportgenerator.graph.latencyVsRequest.title=Latencies Vs Request
jmeter.reportgenerator.graph.latencyVsRequest.exclude_controllers=true
jmeter.reportgenerator.graph.latencyVsRequest.property.set_granularity=${jmeter.reportgenerator.overall_granularity}
jmeter.properties
#---------------------------------------------------------------------------
# Results file configuration
#---------------------------------------------------------------------------
# This section helps determine how result data will be saved.
# The commented out values are the defaults.
# legitimate values: xml, csv, db. Only xml and csv are currently supported.
jmeter.save.saveservice.output_format=csv
# The below properties are true when field should be saved; false otherwise
#
# assertion_results_failure_message only affects CSV output
jmeter.save.saveservice.assertion_results_failure_message=true
#
# legitimate values: none, first, all
jmeter.save.saveservice.assertion_results=all
#
jmeter.save.saveservice.data_type=true
jmeter.save.saveservice.label=true
jmeter.save.saveservice.response_code=true
# response_data is not currently supported for CSV output
jmeter.save.saveservice.response_data=true
# Save ResponseData for failed samples
jmeter.save.saveservice.response_data.on_error=false
jmeter.save.saveservice.response_message=true
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.thread_name=true
jmeter.save.saveservice.time=true
jmeter.save.saveservice.subresults=true
jmeter.save.saveservice.assertions=true
jmeter.save.saveservice.latency=true
# Only available with HttpClient4
#jmeter.save.saveservice.connect_time=true
jmeter.save.saveservice.samplerData=true
#jmeter.save.saveservice.responseHeaders=false
#jmeter.save.saveservice.requestHeaders=false
#jmeter.save.saveservice.encoding=false
jmeter.save.saveservice.bytes=true
# Only available with HttpClient4
jmeter.save.saveservice.sent_bytes=true
jmeter.save.saveservice.url=true
jmeter.save.saveservice.filename=false
jmeter.save.saveservice.hostname=false
jmeter.save.saveservice.thread_counts=true
jmeter.save.saveservice.sample_count=false
jmeter.save.saveservice.idle_time=true
# Timestamp format - this only affects CSV output files
# legitimate values: none, ms, or a format suitable for SimpleDateFormat
#jmeter.save.saveservice.timestamp_format=ms
#jmeter.save.saveservice.timestamp_format=yyyy/MM/dd HH:mm:ss.SSS
# For use with Comma-separated value (CSV) files or other formats
# where the fields' values are separated by specified delimiters.
# Default:
#jmeter.save.saveservice.default_delimiter=,
# For TAB, one can use:
#jmeter.save.saveservice.default_delimiter=\t
# Only applies to CSV format files:
# Print field names as first line in CSV
#jmeter.save.saveservice.print_field_names=true
# Optional list of JMeter variable names whose values are to be saved in the result data files.
# Use commas to separate the names. For example:
#sample_variables=SESSION_ID,REFERENCE
# N.B. The current implementation saves the values in XML as attributes,
# so the names must be valid XML names.
# By default JMeter sends the variable to all servers
# to ensure that the correct data is available at the client.
# Optional xml processing instruction for line 2 of the file:
# Example:
#jmeter.save.saveservice.xml_pi=<?xml-stylesheet type="text/xsl" href="../extras/jmeter-results-detail-report.xsl"?>
# Default value:
#jmeter.save.saveservice.xml_pi=
# Prefix used to identify filenames that are relative to the current base
#jmeter.save.saveservice.base_prefix=~/
# AutoFlush on each line written in XML or CSV output
# Setting this to true will result in less test results data loss in case of Crash
# but with impact on performances, particularly for intensive tests (low or no pauses)
# Since JMeter 2.10, this is false by default
#jmeter.save.saveservice.autoflush=false
So basically facing issue at two places:
How to get the latency value?
When i provide Ramp up value as 1, the script with Thread Group =50 takes around 16 seconds to complete the upload, whereas if i give Ramp up something other than 1 such as 10 then the script ends after 10 secs exact, irrespective of file is getting uploaded or not and providing vague results in html report as well.
Any idea how to solve this. Or need to do anything else in script.
You cannot as the plugin you're using doesn't call SampleResult.setLatency() function anywhere
theoretically it should be possible to request the functionality from the plugin developers
Setting 10 seconds ramp-up period for 50 virtual users means that JMeter starts with 1 virtual user and gradually increases the load to 50 within 10 seconds duration. Make sure to have enough loops defined in the Thread Group as you may run into the situation when 1st user has already finished uploading the file and was terminated and 2nd hasn't need started so you have maximum 1 user concurrency (it can be checked using Active Threads Over Time listener). See JMeter Test Results: Why the Actual Users Number is Lower than Expected for more detailed explanation if needed.
I'm trying to set up goaccess to analyse some apache output which is highly customised. I didn't fancy my chances writing a .goaccessrc file straight off, so decided to simplify the log (in a text editor) and start slowly. However, I can't even get this trivial example to work. I've also tried some examples from SO that are marked as 'Answered', but I'm still getting the rather terse 'Nothing valid to process' message.
Here's a line from my simplified log file:
2014-05-14 06:26:18 "GET / HTTP/1.1" 200 37.157.246.146
and here's my .goaccessrc:
date_format %Y-%m-%d %H:%M:%S
log_format %d "%r" %s %h
I'm sure the .goaccessrc file is in the right place and being read, because if I remove it, I get the Log Format Configuration window when running goaccess. I'm sure it's something trivial, but I just can't see it. Here's the full output of my recent terminal session:
[root#dev ~] # cat .goaccessrc
date_format %Y-%m-%d %H:%M:%S
log_format %d "%r" %s %h
[root#dev ~] # cat /var/log/apache2/simple.log
2014-05-14 06:26:18 "GET / HTTP/1.1" 200 37.157.246.146
[root#dev ~] # goaccess -f /var/log/apache2/simple.log
GoAccess - version 0.7.1 - Apr 18 2014 21:28:20
An error has occurred
Error occured at: goaccess.c - render_screens - 456
Message: Nothing valid to process.
OK, see here for the full answer. It basically boils down to this. All parsing seems to be driven by log_format, and the token separator is the space character. So in the example above, the first %d placeholder in log_format matches up to the end of 2014-05-14 and then stops. Then the next token ("%r") then fails when it finds the beginning of the time portion.
Solution to the above is:
date_format %Y-%m-%d
log_format %d %^ "%r" %s %h
which matches the date (only, not time), then ignores everything up to the first " character, then matches the request URL and then finally the status and host address.
Note it seems that unless the date and time are a single token (no whitespace), you can't match the time portion successfully.
I want to make a new logs for my website every 00:00:01 (= each new day).
The problem with rotatelogs is that you can either precise the max size of the logs before creating a new one, or precise the rotation time in seconds. It's not possible to precise an hour of rotation.
They talk about "cronjobs", but I don't get it. Could someone please explain if it's possible, and if so, give an example?
Thank you very much
Following is the help of the rotatelogs utility:
/ # /opt/httpd/bin/rotatelogs --help Incorrect number of arguments
Usage: /opt/httpd/bin/rotatelogs [-l] [-f] {|} [offset minutes from UTC]
Add this:
TransferLog "|/opt/httpd/bin/rotatelogs /some/where 86400"
or
TransferLog "|/opt/httpd/bin/rotatelogs /some/where 5M"
to httpd.conf. The generated name will be /some/where.nnnn where nnnn
is the system time at which the log nominally starts (N.B. if using a
rotation time, the time will always be a multiple of the rotation
time, so you can synchronize cron scripts with it). At the end of each
rotation time or when the file size is reached a new log is started. /
#
According to this section of the manual page for rotatelogs:
rotationtime
The time between log file rotations in seconds. The rotation
occurs at the beginning of this interval. For example, if the
rotation time is 3600, the log file will be rotated at the
beginning of every hour; if the rotation time is 86400, the log
file will be rotated every night at midnight.
setting 86400 as the period will do what you want (new file started at midnight every day).
Also use the -l option if you want "midnight" to be in your local timezone rather than UTC.
The case when a cronjob would be needed is if you wanted a period of one day but at a different time than midnight. More general tools like logrotate (outside apache) are typically used for that.
I have a CGI script which takes about 1 minute to run. Right now Apache only returns results to the browser once the process has finished.
How can I make it show the output like it was run on a terminal?
Here is a example which demonstrates the problem.
I want to see the numbers 1 to 5 appear as they are printed.
I had to disable mod_deflate to have chunk mode working with apache
I did not find another way for my cgi to disable auto encoding to gzip.
There are several factors at play here. To eliminate a few issues, Apache and bash are not buffering any of the output. You can verify with this script:
#!/bin/sh
cat <<END
Content-Type: text/plain
END
for i in $(seq 1 10)
do
echo $i
sleep 1
done
Stick this somewhere that Apache is configured to execute CGI scripts, and test with netcat:
$ nc localhost 80
GET /cgi-bin/chunkit.cgi HTTP/1.1
Host: localhost
HTTP/1.1 200 OK
Date: Tue, 24 Aug 2010 23:26:24 GMT
Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.7l DAV/2
Transfer-Encoding: chunked
Content-Type: text/plain
2
1
2
2
2
3
2
4
2
5
2
6
2
7
2
8
2
9
3
10
0
When I do this, I see in netcat each number appearing once per second, as intended.
Note that my version of Apache, at least, applies the chunked transfer encoding automatically, presumably because I didn't include a Content-Length; if you return the Transfer-Encoding: chunked header yourself, then you need to encode the output of your script in the chunked transfer encoding. That's pretty easy, even in a shell script:
chunk () {
printf '%x\r\n' "${#1}" # Length of the chunk in hex, CRLF
printf '%s\r\n' "$1" # Chunk itself, CRLF
}
chunk $'1\n' # This is a Bash-ism, since it's pretty hard to get a newline
chunk $'2\n' # character portably.
However, serve this to a browser, and you'll get varying results depending on the browser. On my system, Mac OS X 10.5.8, I see different behaviors between my browsers. In Safari, Chrome, and Firefox 4 beta, I don't start seeing output until I've sent somewhere around 1000 characters (I would guess 1024 including the headers, or something like that, but I haven't narrowed it down to the exact behavior). In Firefox 3.6, it starts displaying immediately.
I would guess that this delay is due to content type sniffing, or character encoding sniffing, which are in the process of being standardized. I have tried to see if I could get around the delay by specifying proper content types and character encodings, but without luck. You may have to send some padding data (which would be pretty easy to do invisibly if you use HTML instead of plain text), to get beyond that initial buffer.
Once you start streaming HTML instead of plain text, the structure of your HTML matters too. Some content can be displayed progressively, while some cannot. For instance, streaming down <div>s into the body, with no styling, works fine, and can display progressively as it arrives. If you try to open a <pre> tag, and just stream content into that, Webkit based browsers will wait until they see the close tag to try to lay that out, while Firefox is happy to display it progressively. I don't know all of the corner cases; you'll have to experiment to see what works for you.
Anyhow, I hope this helps you get started. Let me know if you have any more questions!