I have a table of which one-row cell has so much data that it could span multiple pages in the finally-generated PDF file. rst2pdf ungracefully fails when I feed it my file, with the following output:
[ERROR] pdfbuilder.py:161 Failed to build doc
Traceback (most recent call last):
File "/home/pwng/.local/lib/python3.9/site-packages/rst2pdf/pdfbuilder.py", line 158, in write
docwriter.write(doctree, destination)
File "/usr/lib/python3/dist-packages/docutils/writers/__init__.py", line 78, in write
self.translate()
File "/home/pwng/.local/lib/python3.9/site-packages/rst2pdf/pdfbuilder.py", line 697, in translate
createpdf.RstToPdf(
File "/home/pwng/.local/lib/python3.9/site-packages/rst2pdf/createpdf.py", line 689, in createPdf
pdfdoc.multiBuild(elements)
File "/usr/lib/python3/dist-packages/reportlab/platypus/doctemplate.py", line 1167, in multiBuild
self.build(tempStory, **buildKwds)
File "/usr/lib/python3/dist-packages/reportlab/platypus/doctemplate.py", line 1080, in build
self.handle_flowable(flowables)
File "/home/pwng/.local/lib/python3.9/site-packages/rst2pdf/createpdf.py", line 859, in handle_flowable
self.handle_frameEnd()
File "/usr/lib/python3/dist-packages/reportlab/platypus/doctemplate.py", line 726, in handle_frameEnd
self.handle_pageEnd()
File "/usr/lib/python3/dist-packages/reportlab/platypus/doctemplate.py", line 668, in handle_pageEnd
raise LayoutError(ident)
reportlab.platypus.doctemplate.LayoutError: More than 10 pages generated without content - halting layout. Likely that a flowable is too large for any frame.
FAILED
build succeeded.
and make latexpdf produce undesirable output depicted in the following screenshot.
Is there a way to remedy this problem using either latexpdf or rst2pdf? Ideally, I would like a solution that works for both spaced text (i.e. space-separated words) and consecutive, wrapped non-separated text.
This isn't the answer you want, but rst2pdf won't split the cell across pages, so if it doesn't fit onto the page, it won't be able to generate the document. The project (I'm a maintainer) is open to patches, in case you end up fixing it yourself. I'm not aware of a workaround, other than reformatting the content to be more printable.
Related
I met a problem.
I run the command "gitbook pdf", and it show me this Error:
EbookError: Error during ebook generation: 1% Converting input to HTML...
InputFormatPlugin: HTML Input running
on /tmp/tmp-72206AoHgIzetV6D/SUMMARY.html
Language not specified
Creator not specified
Building file list...
Normalizing filename cases
Rewriting HTML links
34% Running transforms on ebook...
Merging user specified metadata...
Detecting structure...
Detected chapter: Introduction
Detected chapter: CH1
Detected chapter: CH2
Detected chapter: Item 1
Auto generated TOC with 4 entries.
Flattening CSS and remapping font sizes...
Failed to parse CSS selector: u'h|button::-moz-focus-inner'
Failed to parse CSS selector: u'h|input::-moz-focus-inner'
Failed to parse CSS selector: u'h|input[type=search]::-webkit-search-cancel-button'
Failed to parse CSS selector: u'h|input[type=search]::-webkit-search-decoration'
Source base font size is 12.00000pt
Removing fake margins...
Cleaning up manifest...
Trimming unused files from manifest...
Creating PDF Output...
67% Running PDF Output plugin
Traceback (most recent call last):
File "/usr/bin/ebook-convert", line 20, in<br>
sys.exit(main())
File "/usr/lib/calibre/calibre/ebooks/conversion/cli.py", line 359, in main<br>
plumber.run()
File "/usr/lib/calibre/calibre/ebooks/conversion/plumber.py", line 1189, in run
self.opts, self.log)
File "/usr/lib/calibre/calibre/ebooks/conversion/plugins/pdf_output.py", line 123, in convert
must_use_qt()
File "/usr/lib/calibre/calibre/gui2/__init__.py", line 1041, in must_use_qt
raise RuntimeError('X server required. If you are running on a'
RuntimeError: X server required. If you are running on a headless machine, use xvfb
what's the error mean?
Summary of a problem:
I am writing some code that checks the content of a FTS file header (data saved from a telescope) using astropy.io.fits. My problem is when I try to open .FTS.gz files instead of .FTS files on a remote server. When I open() a .FTS.gz I get errors, if I gunzip the .FTS.gz file, all is good. One of the errors suggest I have an END missing card. Searching online, I used a suggestion of using the ignore_missing_end=True argument in fits.open(), but then I get the next error. This next error suggests my FITS file is empty or corrupt, however it is not the case. I can open it with SAOImage DS9 without any problems, plus I have run this handy online tool called fitsverify which reports no errors in my file. If I download the offending file .FTS.gz and run a similar code to fits.open() this file locally, I get no errors at all. An example of an offending file (used in the code below) is now uploaded here.
The Astropy documentation says:
"Working with compressed files
The open() function will seamlessly open FITS files that have been compressed with gzip, bzip2 or pkzip. Note that in this context we’re talking about a fits file that has been compressed with one of these utilities - e.g. a .fits.gz file."
How do I open a remote .FTS.gz file without downloading it? I have hundreds of thousands of files like this, so downloading is not an option and it is not just one file that gives a problem, it is all of them.
Thanks,
Aina.
Code and errors:
CODE TO OPEN A REMOTE .FTS.gz FILE:
from astropy.io import fits
import paramiko
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.load_system_host_keys()
client.connect('myhostname', username='myusername', password='mypassword')
apath = '/path/to/folder/to/search'
apattern = '"RUN0001.FTS.gz"'
rawcommand = 'find {path} -name {pattern}'
command = rawcommand.format(path=apath, pattern=apattern)
stdin, stdout, stderr = client.exec_command(command)
filelist = stdout.read().splitlines()
for i in filelist:
sftp_client = client.open_sftp()
remote_file = sftp_client.open(i)
hdulist = fits.open(remote_file)
client.close()
ERROR:
Traceback (most recent call last):
File "/Users/amusaeva/Documents/PyCharm/FITSHeaders/stackoverflow.py", line 17, in <module>
hdulist = fits.open(remote_file)
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 166, in fitsopen
lazy_load_hdus, **kwargs)
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 404, in fromfile
lazy_load_hdus=lazy_load_hdus, **kwargs)
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1040, in _readfrom
read_one = hdulist._read_next_hdu()
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1135, in _read_next_hdu
hdu = _BaseHDU.readfrom(fileobj, **kwargs)
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/base.py", line 329, in readfrom
**kwargs)
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/base.py", line 394, in _readfrom_internal
header = Header.fromfile(data, endcard=not ignore_missing_end)
File "/Library/Python/2.7/site-packages/astropy/io/fits/header.py", line 450, in fromfile
padding)[1]
File "/Library/Python/2.7/site-packages/astropy/io/fits/header.py", line 519, in _from_blocks
raise IOError('Header missing END card.')
IOError: Header missing END card.
Process finished with exit code 1
CHANGING THE CODE ABOVE FOR ONE LINE ONLY:
hdulist = fits.open(remote_file, ignore_missing_end=True)
ERROR:
WARNING: VerifyWarning: Error validating header for HDU #0 (note: Astropy uses zero-based indexing).
Header size is not multiple of 2880: 7738429
There may be extra bytes after the last HDU or the file is corrupted. [astropy.io.fits.hdu.hdulist]
Traceback (most recent call last):
File "/Users/amusaeva/Documents/PyCharm/FITSHeaders/stackoverflow.py", line 17, in <module>
hdulist = fits.open(remote_file, ignore_missing_end=True)
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 166, in fitsopen
lazy_load_hdus, **kwargs)
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 404, in fromfile
lazy_load_hdus=lazy_load_hdus, **kwargs)
File "/Library/Python/2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1044, in _readfrom
raise IOError('Empty or corrupt FITS file')
IOError: Empty or corrupt FITS file
Process finished with exit code 1
CODE TO OPEN THE OFFENDING .FTS.gz FILE LOCALLY PRODUCES NO ERRORS:
import os
from astropy.io import fits
folderTosearch = "/path/to/folder/to/search/locally";
for root, dirs, files in os.walk(folderTosearch):
for file in files:
if file.endswith("RUN0001.FTS.gz"):
hdulist = fits.open(os.path.join(root, file))
This happens because the sftp call passes some variant of a file-like object (which has a .read() method that fits.open() will use.
The file like object, however, is still a gzip file. Astropy checks whether a file is zipped only for file names, that is, when the argument to fits.open() is a string (that happens to be a path). Astropy does not appear to test for the magic bytes that identify a byte stream as a gzip file. Oddly enough, it does do this verification when path strings are passed. Arguably, this may be a slight shortcoming in the astropy.io.fits module, but perhaps there's a reason for it.
(Disclaimer: the above conclusion is from scanning quickly through the relevant source code; I may have missed something. Hopefully people will correct me if so.)
One solution is to do the unzipping yourself. I've cobbled up the following:
from cStringIO import StringIO
import zlib
<...>
for i in filelist:
sftp_client = client.open_sftp()
remote_file = sftp_client.open(i)
decompressed = StringIO(
zlib.decompress(remote_file.read(), zlib.MAX_WBITS|32))
hdulist = fits.open(decompressed)
client.close()
Above, we're reading the full contents of the remote file (remote_file.read(), then uncompressing the contents. That results in a string, so we wrap it in a StringIO instance to make it a file-like object again, that we can pass to fits.open(). (For the zlib.MAX_WBITS|32 argument: see this answer.)
Alternatively, you can sftp the file to local disk, and then read the file (with the local filename) locally. The above just keeps everything in memory.
I too have run across the infamous pyglet memory leak, that occurs when one needs to update text components at the frame level (with a text variable typically computed in a separate code component, also at frame level).
The leak/bug crashes a running script with an error message of the sort:
File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\psychopy-1.83.04-py2.7.egg\psychopy\visual\window.py", line 541, in flip
thisStim.draw()
File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\psychopy-1.83.04-py2.7.egg\psychopy\visual\text.py", line 603, in draw
self._pygletTextObj.draw()
File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pyglet\font\__init__.py", line 565, in draw
self._layout.draw()
File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pyglet\text\layout.py", line 852, in draw
self.batch.draw()
File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pyglet\graphics\__init__.py", line 544, in draw
func()
File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pyglet\graphics\__init__.py", line 476, in <lambda>
(lambda d, m: lambda: d.draw(m))(domain, mode))
File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pyglet\graphics\vertexdomain.py", line 313, in draw
glDrawArrays(mode, starts[0], sizes[0])
WindowsError: exception: access violation reading 0x00000010
Exception TypeError: "'NoneType' object is not callable" in <bound method Server.__del__ of <pyolib.server.Server object at 0x0EFB3670>> ignored
I know the PsychoPy community has linked this to a memory leak in pyglet that was meant to have been fixed around April 2016 according to this thread. I downloaded the latest version of Pyglet (which is v1.2.4 - released apparently in Sep 2015, so before the supposed fix of the leak) and replaced its contents under c:\Program Files (x86)\PsychoPy2\Lib\site-packages\pyglet\. However, this did not fix my problem.
Unfortunately, my experiment does not make sense if I give up the frame-wise computation that feeds the text component.
An older thread that I found suggested the following workaround: in the code component that computes frame-wise the value of the text variable being displayed, only assign this variable a new value if that new value actually differs from the previous value. However, even after changing my code component this way, the pyglet error still pops up.
Assuming the bug was fixed in Pyglet, is it not v1.2.4 that reflects this fix, and should it not be updated by overwriting the contents of c:\Program Files (x86)\PsychoPy2\Lib\site-packages\pyglet\ (maintaining the original folder structure)?
Assuming the bug was in fact NOT fixed in Pyglet, can anyone suggested a different workaround that I can try to make my PsychoPy expt not crash?
Many thanks indeed for your help.
(Apologies for the double post both here and on the psychopy group; will update both threads as soon as there is a response on the other, to avoid people wasting time with double answers).
One workaround is to use TextBox instead of TextStim. To do that, one first defines a stimulus object in a code component (under "begin routine"), such as
textbox = visual.TextBox(window=win,
text='Something',
font_size=21,
font_color=[-1,-1,1],
size=(1.9, .3),
pos=(0.0, 0.25),
grid_horz_justification='center',
units='norm')
then, perhaps under the "every frame" tab, one uses code to customise & call upon the thus-defined textbox, with syntax as found on http://www.psychopy.org/api/visual/textbox.html
I want to run two instances of OpenERP but once the first one is running I don't know why the second one access to the files of the first one, like ir_http.py from the first one. I noticed that it changes the directory, I did some debug on the files used with no luck because somehow the second instance manages to call the first one, trying to access their directory. This is my log:
File "/home/user/lib/python2.7/Werkzeug-0.9.4-py2.7.egg/werkzeug/wsgi.py", line 579, in __call__
return self.app(environ, start_response)
File "/home/user/openerp-8.0-acmlpsc/openerp-8.0/openerp/http.py", line 1234, in dispatch
result = ir_http._dispatch()
File "/home/user/openerp-8.0/openerp/addons_test/trunk-restaurant-addons/base/ir/ir_http.py", line 106, in _dispatch
werkzeug.exceptions.Forbidden))
File "/home/user/openerp-8.0/openerp/addons_test/trunk-restaurant-addons/base/ir/ir_http.py", line 101, in _dispatch
auth_method = self._authenticate(func.routing["auth"])
File "/home/user/openerp-8.0/openerp/addons_test/trunk-restaurant-addons/base/ir/ir_http.py", line 76, in _authenticate
request.session.check_security()
File "/home/user/openerp-8.0-acmlpsc/openerp-8.0/openerp/http.py", line 801, in check_security
security.check(self.db, self.uid, self.password)
I can understand that the security check fails, because is the second instance trying to access the first one but why is the second one trying to access the first one! I don't get it.
It appears that you have two users (openerp-8.0-acmlpsc & openerp-8.0), one for each instance of OpenERP. That is correct!
Make sure that each of those users are the owners of each of their own directories (server, addons, etc)
Also make sure that each of those servers are using different configuration files. A lot of setups will put one /etc/openerp-server.conf and another in /etc/openerp-server-test.conf
They should be using different ports and own their own databases.
I'm using fileinput to read some large data:
import gzip
import fileinput
f=gzip.open('/scratch/try.fastq.gz','r')
for line in fileinput.input(f):
print line
However I got errors like:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/share/lib/python2.6/fileinput.py", line 253, in next
line = self.readline()
File "/share/lib/python2.6/fileinput.py", line 345, in readline
self._file = open(self._filename, self._mode)
IOError: [Errno 2] No such file or directory: '#HWI-ST150_0129:2:1:13466:2247#0/1\n'
Cannot fileinput take file object as input? Then how to use fileinput to deal with gzip file?
thx
Nope, the first argument to fileinput.input should be a list of filenames. What you want can be achieved with
for line in gzip.open('/scratch/try.fastq.gz')
print line
fileinput exists to support the idiom where a program reads from a list of files, probably supplied on the command line, or standard input if no files have been specified. If you still want to use it, even though it's useless in your example, you should do
for line in fileinput(['/scratch/try.fastq.gz'], openhook=gzip.open):
print line
As other sources have said, the value for openhook must be a function, but that doesn't mean you can't call a function to return a function. For example, if you want to support multiple different types of incoming files you could write something like this:
import fileinput
import gzip
def get_open_handler(compressed):
if deciding_data:
# mode comes in as 'r' by defualt, but that means binary to `gzip`
return lambda file_name, mode: gzip.open(file_name, mode='rt')
else:
# the default mode of 'r' means text for `open`
return open
# get args here
for line in fileinput.input(args.files, openhook=get_open_handler(args.compressed))
print(line)
As you can see, we are calling a function from openhook, but that function returns another function. In this case, we are fixing the mode of gzip.open, but we can do anything we want, including using functools.partial to bind some values to a function so that when the default filename and mode get passed to the function assigned to openhook, the function will do what you want.