Using -log-level 4 on x265 v1.4 I get encoding time and elapsed time per frame.
Now using v1.7 I don't get these values in csv. Instead I get DecideWait (ms), Row0Wait (ms), Wall time (ms), Ref Wait Wall (ms), Total CTU time (ms), Stall Time (ms), Avg WPP, amd Row Blocks.
I tried -log-level values 3,4 and 5 and still no luck.
can you help me please?
I found the solution.
x265 v2.5 now gives the Total frame time (ms) in the csv log file.
When you run an x265 encoding add the following flag and csv created file location:
--csv-log-level 2 --csv info.csv
More info can be found here and here.
For x265 earlier versions, a rough estimation of the elapsed time per frame is calculated by the summation of DecideWait (ms), Row0Wait (ms), and Wall time (ms).
Related
I tried building a program that scans PDF files to find certain elements and then outputs a new PDF file with all the pages that contain the elements. It was originally single-threaded and a bit slow to run. It took about 36 seconds on a 6-Core-5600X. So I tried multiprocessing it with concurrent.futures:
def process_pdf(filename):
# Open the PDF file
f = open(filename, "rb")
print("Searching: " + filename)
# Create a PDF object
pdf = PyPDF2.PdfReader(f)
# Extract the text from each page in the PDF
extracted_text = [page.extract_text() for page in pdf.pages]
# Initialize a list
matching_pages = []
# Iterate through the extracted text
for j, text in enumerate(extracted_text):
# Search for the symbol in the text
if symbol in text:
# If the symbol is found, get a new PageObject instance for the page
page = pdf.pages[j]
# Add the page to the list
matching_pages.append(page)
return matching_pages
Multiprocessing Block:
with concurrent.futures.ThreadPoolExecutor() as executor:
# Get a list of the file paths for all PDF files in the directory
file_paths = [
os.path.join(directory, filename)
for filename in os.listdir(directory)
if filename.endswith(".pdf")
]
futures = [executor.submit(process_pdf, file_path) for file_path in file_paths]
# Initialize a list to store the results
results = []
# Iterate through the futures as they complete
for future in concurrent.futures.as_completed(futures):
# Get the result of the completed future
result = future.result()
# Add the result to the list
results.extend(result)
# Add each page to the new PDF file
for page in results:
output_pdf.add_page(page)
The multiprocessing works, as evident from the printed text, but it somehow doesn't scale at all. 1 thread ~ 35 seconds, 12 Threads ~ 38 seconds.
Why? Where's my bottleneck?
Tried using other libraries, but most were broken or slower.
Tried using re instead of in to search for the symbol, no improvement.
In general PDF consumes much of a devices primary resources, Each device and system will be different so by way of illustration here is one simple PDF task.
Search for text by a common python library component poppler pdftotext, (others may be faster but the aim here is to show the normal attempts and issues)
As a ballpark yardstick for 1 minute I scanned roughly 2500 pages of one file for the word "AMENDMENTS" found 900 occurrences such as
page 82
[1] SHORT TITLE OF 1971 AMENDMENTS
[18] SHORT TITLE OF 1970 AMENDMENTS
[44] SHORT TITLE OF 1968 AMENDMENTS
[54] SHORT TITLE OF 1967 AMENDMENTS
page 83
[11] SHORT TITLE OF 1966 AMENDMENTS
[23] SHORT TITLE OF 1965 AMENDMENTS
[42] SHORT TITLE OF 1964 AMENDMENTS
page 84
[16] SHORT TITLE OF 1956 AMENDMENTS
[26] SHORT TITLE OF 1948 AMENDMENTS
page 85
[43] DRUG ABUSE, AND MENTAL HEALTH AMENDMENTS ACT OF 1988
to scan the whole file (13,234 pages) would be about 5 mins 20 seconds
and I know from testing 4 cpus could process 1/4 of that file (3,308 pages from 13,234) in under 1 minute (there is a gain for using smaller files).
So a 4 core device should do say 3000 pages per core in a short time, well lets see.
If I thread that as 3 x 1000 pages one finishes after about 50 seconds another about 60 seconds and a 3rd about 70 seconds
overall there is little or just a slight gain, so the cause is one application spending one third of its time in each thread.
Overall about 3000 pages per minute
Lets try clever lets use 3 applications on the one file. Surely they can each take less time, guess what
one finishes after about 50 seconds another about 60 seconds and a 3rd about 70 seconds no gain using 3 applications
Overall about 3000 pages per minute
Lets try more clever lets use 3 applications on 3 similar but different files. Surely they can each take less time, guess what
one finishes after about 50 seconds another about 60 seconds and a 3rd about 70 seconds no gain using 3 applications with 3 tasks
Overall about 3000 pages per minute
Whatever way I try the resources on this device are constrained to Overall about 3000 pages per minute.
I may just as well let 1 thread run un-fettered.
So the basic answer is use multiple devices, same as graphics farming, is done.
Hallo to all userse and helpers here in this forum! Thank you, i am new and i have found allready a lot of solutions.
Now I want to ask, if someone can help me:
I have a Movie about 30 seconds made of 1 image.
Now I want to pixelate depending on time or framenumber - every time a litlle bit less.
My code so far:
ffmpeg -i in.mp4 -vf scale=iw/n:ih/n,scale=niw:nih:flags=neighbor out.mp4
where n should be the framenumber 1 to 900.
this scould also be t+1 for slower change.
the stars are gone - so i mean n times iw:n times ih:
error-massage:
undefined constant or missing '(' in 'n'
error when evaluating the expression 'ih/n'
maybe the expression for out_w:'w/n' or for out_h:'ih/n' is self-referencing.
failed to configure output pad on paresed_scale_0
error reinitializing filters!
failed to inject frame into filter network: invalid argument
error while processing the decoded data for stream #0:0
Do you have some suggestion plaese - Thank you in Advance
Having updated to 7.9.0 and after initial problems: https://stackoverflow.com/questions/51597744/optaplanner-7-9-0-and-adding-multithreading-same-planningid-exception I now have been trying to test and compare to my 7.7.0 version. However I cannot get it to reproduce the same solution everytime (obviously only with the same problem data) as in my older version, even when explicitly set in the config xml. Is there additional set up for this version required?
Edit: Did some testing and found switching to TABU (haven't been through them all) gave me the expected consistency:
Extra Cores:1 - DEFAULT:
Score (Hrd:Med:Sft) Time Taken (Minutes:Seconds)
Test1 0:0:-7609 0:08
Test2 0:-1:-7758 0:13
Test3 0:-1:-7705 0:14
Extra Cores:1 - TABU:
Score (Hrd:Med:Sft) Time Taken (Minutes:Seconds)
Test1 0:0:-7763 1:29
Test2 0:0:-7763 1:29
Test3 0:0:-7763 1:28
Between two runs of the former the solution diverges at LS step 28:
LS step (25), time spent (1869), score (0hard/-3medium/-8155soft),
LS step (26), time spent (1890), score (0hard/-3medium/-8339soft),
LS step (27), time spent (1895), score (0hard/-3medium/-8126soft),
**LS step (28), time spent (1909), score (0hard/-3medium/-8256soft),
LS step (29), time spent (1915), score (0hard/-3medium/-8438soft),
LS step (30), time spent (1924), score (0hard/-3medium/-8620soft),
LS step (31), time spent (1952), score (0hard/-3medium/-8639soft),**
...and...
LS step (25), time spent (1385), score (0hard/-3medium/-8155soft),
LS step (26), time spent (1407), score (0hard/-3medium/-8339soft),
LS step (27), time spent (1412), score (0hard/-3medium/-8126soft),
**LS step (28), time spent (1422), score (0hard/-3medium/-8217soft),
LS step (29), time spent (1436), score (0hard/-3medium/-8336soft),
LS step (30), time spent (1442), score (0hard/-3medium/-8517soft),
LS step (31), time spent (1448), score (0hard/-3medium/-8571soft),**
Don't know if that all makes it more or less likely to be problem/solution set up or something else.
First, let's define reproducible: getting the same result at the same step iteration. If the CPU time is completely the same (which it never is), this also means getting the same result after the same amount of time. So you might need to run it a bit longer to get the same amount of steps.
Multithreaded solving is reproducible if and only if the same moveThreadCount is used. A run with <moveThreadCount>4</> is NOT reproducible by a <moveThreadCount>2</>. Even a run with <moveThreadCount>NONE</> is not reproducible with a run with <moveThreadCount>1</> (the latter being useless except for QA btw).
Look at your score calculation speed.
If it's lower, you probably forgot to increase the memory size and GC churns are hurting you.
If that's higher, you'd normally get a better result. If you see a worse result anyway, the config might have been overfitted for the no move threads. Use additional datasets in optaplanner-benchmark to prove or disprove that.
Furthermore, if you use <moveThreadCount>AUTO</>, it's only reproducible on the same machine, because AUTO changes the moveThreadCount depending on the number of CPU cores in the machine.
I want to make a new logs for my website every 00:00:01 (= each new day).
The problem with rotatelogs is that you can either precise the max size of the logs before creating a new one, or precise the rotation time in seconds. It's not possible to precise an hour of rotation.
They talk about "cronjobs", but I don't get it. Could someone please explain if it's possible, and if so, give an example?
Thank you very much
Following is the help of the rotatelogs utility:
/ # /opt/httpd/bin/rotatelogs --help Incorrect number of arguments
Usage: /opt/httpd/bin/rotatelogs [-l] [-f] {|} [offset minutes from UTC]
Add this:
TransferLog "|/opt/httpd/bin/rotatelogs /some/where 86400"
or
TransferLog "|/opt/httpd/bin/rotatelogs /some/where 5M"
to httpd.conf. The generated name will be /some/where.nnnn where nnnn
is the system time at which the log nominally starts (N.B. if using a
rotation time, the time will always be a multiple of the rotation
time, so you can synchronize cron scripts with it). At the end of each
rotation time or when the file size is reached a new log is started. /
#
According to this section of the manual page for rotatelogs:
rotationtime
The time between log file rotations in seconds. The rotation
occurs at the beginning of this interval. For example, if the
rotation time is 3600, the log file will be rotated at the
beginning of every hour; if the rotation time is 86400, the log
file will be rotated every night at midnight.
setting 86400 as the period will do what you want (new file started at midnight every day).
Also use the -l option if you want "midnight" to be in your local timezone rather than UTC.
The case when a cronjob would be needed is if you wanted a period of one day but at a different time than midnight. More general tools like logrotate (outside apache) are typically used for that.
I am benchmarking php application with apache benchmark. I have the server on my local machine. I run the following:
ab -n 100 -c 10 http://my-domain.local/
And get this:
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 3 3.7 2 8
Processing: 311 734 276.1 756 1333
Waiting: 310 722 273.6 750 1330
Total: 311 737 278.9 764 1341
However, if I refresh my browser on the page http://my-domain.local/ I find out it takes a lot longer than 737 ms (the mean that ab reports) to load the page (around 3000-4000 ms). I can repeat this many times and the loading of the page in the browser always takes at least 3000 ms.
I tested another, heavier page (page load in browser takes 8-10 seconds). I used a concurrency of 1 to simulate one user loading the page:
ab -n 100 -c 1 http://my-domain.local/heavy-page/
And the results are here:
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 17 20 4.7 18 46
Waiting: 16 20 4.6 18 46
Total: 17 20 4.7 18 46
So what does the total line on the ab results actually tell? Clearly it's not the number of milliseconds the browser is loading the web page. Is the number of milliseconds that it takes from browser to load the page (X) linearly dependent of number of the total mean milliseconds ab reports (Y)? So if I'm able to reduce half of Y, have I also reduced half of X?
(Also Im not really sure what Processing, Waiting and Total mean).
I'll reopen this question since I'm facing the problem again.
Recently I installed Varnish.
I run ab like this:
ab -n 100 http://my-domain.local/
Apache bench reports very fast response times:
Requests per second: 462.92 [#/sec] (mean)
Time per request: 2.160 [ms] (mean)
Time per request: 2.160 [ms] (mean, across all concurrent requests)
Transfer rate: 6131.37 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1 2 2.3 1 13
Waiting: 0 1 2.0 1 12
Total: 1 2 2.3 1 13
So the time per request is about 2.2 ms. When I browse the site (as an anonymous user) the page load time is about 1.5 seconds.
Here is a picture from Firebug net tab. As you can see my browser is waiting 1.68 seconds for my site to response. Why is this number so much bigger than the request times ab reports?
Are you running ab on the server? Don't forget that your browser is local to you, on a remote network link. An ab run on the webserver itself will have almost zero network overhead and report basically the time it takes for Apache to serve up the page. Your home browser link will have however many milliseconds of network transit time added in, on top of the basic page-serving overhead.
Ok.. I think I know what's the problem. While I have been measuring the page load time in browser I have been logged in.. So none of the heavy stuff is happening. The page load times in browser with anonymous user are closer to the ones ab is reporting.