When writing POD documentation, I realized that Unicode character ŷ became X on output.
Input:
=pod
=encoding utf8
=over
=item I<yt> (ŷ(t))
The value predicted for time I<t>.
=back
Output in PuTTY:
Input in Emacs:
Version of perldoc being used that of Perl 5.18.2 (SLES12 SP4, perl-5.18.2-12.20.1.x86_64), and LANG=en_US.UTF-8.
Update:
It seems to be a bug in Perl or in the package of SLES12 SP4: Using the same test on OpenSUSE Leap 15.1 with Perl 5.26.1, the output looks OK:
yt (ŷ(t))
The value predicted for time t.
However using pod2man from perl-5.26.1-15.87.x86_64 of openSUSE Leap 15.3, the output is not correct.
OTOH using perldoc the output is correct, too.
Related
I wish to read byte sequences that will not decode as valid UTF-8, specifically byte sequences that correspond to high and low surrogates code points. The result should be a raku string.
I read that, in raku, the 'utf8-c8' encoding can be used for this purpose.
Consider code point U+D83F. It is a high surrogate (reserved for the high half of UTF-16 surrogate pairs).
U+D83F has a byte sequence of 0xED 0xA0 0xBF, if encoded as UTF-8.
Slurping a file? Works
If I slurp a file containing this byte sequence, using 'utf8-c8' as the encoding, I get the expected result:
echo -n $'\ud83f' >testfile # Create a test file containing the byte sequence
myprog1.raku:
#!/usr/local/bin/raku
$*OUT.encoding('utf8-c8');
print slurp('testfile', enc => 'utf8-c8');
$ ./myprog1.raku | od -An -tx1
ed a0 bf
✔️ expected result
Slurping a filehandle? Doesn't work
But if I switch from slurping a file path to slurping a filehandle, it doesn't work, even though I set the filehandle's encoding to 'utf8-c8':
myprog2.raku
#!/usr/local/bin/raku
$*OUT.encoding('utf8-c8');
my $fh = open "testfile", :r, :enc('utf8-c8');
print slurp($fh, enc => 'utf8-c8');
#print $fh.slurp; # I tried this too: same error
$ ./myprog2.raku
Error encoding UTF-8 string: could not encode Unicode Surrogate codepoint 55359 (0xD83F)
in block <unit> at ./myprog2.raku line 4
Environment
Edit 2022-10-30: I originally used my distro's package (Fedora Linux 36: Rakudo version 2020.07). I just downloaded the latest Rakudo binary release (2022.07-01). Result was the same.
$ /usr/local/bin/raku --version
Welcome to Rakudo™ v2022.07.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2022.07.
$ uname -a
Linux hx90 5.19.16-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Oct 16 22:50:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: Fedora
Description: Fedora release 36 (Thirty Six)
Release: 36
Codename: ThirtySix
Querying DB2 from python using ODBC, I am seeing NULL values converted to 0 (on Linux, seemingly corrupt but close to 0 on Mac M1 -- even more worryingly).
This is using the db2 docker image started like this:
docker run -itd --name db2 --privileged=true -p 50000:50000 -e LICENSE=accept -e DB2INST1_PASSWORD=xxxxx -e DBNAME=testdb -v <db storage dir>:/database ibmcom/db2
Code as follows recreates the issue:
import pyodbc
cs = "Driver={ODBC Driver v11.5.7 for DB2};Database=xxxxx;Hostname=xxxx;Port=50000;Protocol=TCPIP;Uid=xxxx;Pwd=xxxx;"
cnxn = pyodbc.connect(cs)
crsr = cnxn.cursor()
crsr.execute("SELECT CAST(NULL AS INT), CAST(NULL AS REAL) FROM SYSIBM.SYSDUMMY1");
print(crsr.messages)
print(crsr.fetchall())
Outputs:
❯ python float-test.py
[]
[(2, 4.2439915814e-314)]
Is it expected that I can't retrieve NULL values as plain data types? It seems to be allowed in PostgreSQL. I know I can cast around this but would rather not, obviously.
Extra Info
It does seem that the ODBC driver version 11.5.7 from Fix Central suffers this issue whilst the 11.5.6 version from https://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli does not.
As mentioned in comments, it appears pyodbc is impacted and both plain ibm_db and DBI are not impacted (both return None, None). So at least there is a workaround.
The reason for the behaviour deifference is that pyodbc is using SQLGetData() while the other two use the SQLBindCol() methods of extracting the result set data.
IBM's clidriver on Linux x64, SQLGetData() sets the (SQLLEN *) StrLen_Or_IndPtr parameter to SQL_NULL_DATA when the value of the column in result-set is NULL. But the problem is that IBMs clidrver sets StrLen_or_IndPtr to SQL_NULL_DATA (as int, 4 bytes), when pyodbc code expects it to SQL_NULL_DATA (as SQLLEN, 8bytes on Linux x64) as SQLLEN is the documented datatype for the StrLen_or_IndPtr argument.
Therefore the comparison in pyodbc getdata.cpp GetDataDouble() :
if ( cbFetched == SQL_NULL_DATA )
Py_RETURN_NONE;
will be false, causing the code to return an unitialised variable instead of Py_None.
I do not know if the maintainers of pyodbc run their tests against a Db2-LUW product, but it looks like other parts of the code could suffer the same problem and other issues may lurk. Consider asking on github what is the support policy for Db2-LUW in pyodbc.
If you have a support contract, IBM should also be asked to comment on their reason for not respecting the datatype of StrLen_Or_IndPtr when writing SQL_NULL_DATA to this parameter on Linux x64.
#!/usr/bin/dyalog -script
⍝ /usr/bin/dyalog is a symlink to /opt/mdyalog/18.0/64/unicode/mapl
factors←{⎕ML ⎕IO←1 ⋄ ⍵{ ⍵,(⍺÷×/⍵)~1}∊⍵{(0=(⍵*⍳⌊⍵⍟⍺)|⍺)/⍵}¨⍬{nxt←⊃⍵ ⋄ msk←0≠nxt|⍵ ⋄ ∧/1↓msk:⍺,⍵ ⋄ (⍺,nxt)∇ msk/⍵}⍵{ (0=⍵|⍺)/⍵}2,(1+2×⍳⌊0.5×⍵*÷2),⍵}
factors 20
Copied from https://dfns.dyalog.com/c_factors.htm
It works exactly as the example apart from the fact I am not able to to type it as separate lines and have to resort to ⋄'s
Using the example it instead results in
./.local/src/sandbox/apl/Main.apl
SYNTAX ERROR
factors←{⎕ML ⎕IO←1 ⍝ Prime factors of ⍵.
Another issue is using ] commands like ]display or ]box on
Using them always results in
./.local/src/sandbox/apl/Main.apl
VALUE ERROR: Undefined name: ⎕SE.UCMD
Try* adding setting DYALOG_LINEEDITOR_MODE to 1:
#!/usr/bin/dyalog -script DYALOG_LINEEDITOR_MODE=1
When running in script mode, SALT, and therefore user commands, are not initialised automatically. As per APLcart, you can enable both with:
(⎕NS⍬).(_←enableSALT⊣⎕CY'salt')
However, under program control, you're usually better off using proper functions than user commands. You can copy in the display and disp functions (which take an array and produce character matrices equivalent to what you'd see from ]display and ]box on) with:
'display' 'disp'⎕CY'dfns'
* Both -script and DYALOG_LINEEDITOR_MODE are experimental in version 18.0, while 18.2 (scheduled for release in March 2022) has dedicated #! script support.
I am trying to take the contents of a file that has a Hex number and convert that number to Binary and output to a file.
This is what I am trying but not getting the binary value:
xxd -r -p Hex.txt > Binary.txt
The contents of Hex.txt is: ff
I have also tried FF and 0xFF, but would like to just use ff since the device I am pulling the info from has it in that format.
Instead of 11111111 which it should be, I get a y with 2 dots above it.
If I change it to ee, I get an i with 2 dots. It seems to be reading it just fine but according to what I have read on the xxd -r -p command, it is not outputing it in the correct format.
The other ways I have found to convert Hex to Binary have either also not worked or is a pretty big Bash script that seems unnecessary to do what I thought would be a simple task.
This also gives me the y with 2 dots.
$ for i in $(cat Hex.txt) ; do printf "\x$i" ; done > Binary.txt
For some reason almost every solution I find gives me this format instead of a human readable Binary value with 1s and 0s.
Any help is appreciated. I am planning on using this in a script to pull the Relay values from Digital Loggers devices using curl and giving Home Assistant a readable file to record the Relay State. Digital Loggers curl cmd gives the state of all 8 relays at once using Hex instead of being able to pull the status of a specific relay.
If "file.txt" contains:
fe
0a
and you run this:
perl -ane 'printf("%08b\n",hex($_))' file.txt
You'll get this:
11111110
00001010
If you use it a lot, you might want to make a bash function of it in your login profile along these lines - being extremely respectful of spaces and semi-colons that might look unnecessary:
bin(){ perl -ane 'printf("%08b\n",hex($_))' $1 ; }
Then you'll be able to do:
bin file.txt
If you dislike Perl for some reason, you can achieve something similar without it as follows:
tr '[:lower:]' '[:upper:]' < file.txt |
while read h ; do
echo "obase=2; ibase=16; $h" | bc
done
This Python code shows how to call some process in Windows 10 and to send to it string commands, to read its string responses through stdin, stdout pipes of the process:
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import *
>>> p = Popen("c:/python38/python.exe", stdin=PIPE, stdout=PIPE)
>>> p.stdin.write(b"print(1+9)\n")
11
>>> p.communicate()
(b'10\r\n', None)
>>>
As you can see the python.exe process returned 10 as an answer to print(1+9). Now I want to do the same in Pharo (or Squeak): in Windows 10 OS - I suppose something similar, i.e. short, simple, understandable, really working.
I installed OSProcess, ProcessWrapper (they were missing in Pharo, also its strange that I got warning that they are not marked for Pharo 8.0 and were not checked to work in Pharo 8.0, but OK), and I tried ProcessWrapper, PipeableOSProcess (copy-pasted different snippets from the Web), etc - with zero success! The results were:
nothing happens, python.exe was not started
VM errors console was opened (white console in the bottom of the Pharo, which is controlled with F2 menu)
different exceptions
etc
Would somebody show me simple working example how to start a process and to to send it commands, read answers, then send again, and so on in some loop - I plan to have such communication in a detached thread and to use it as some service, because Pharo, Smalltalk in general is missing most bindings, so then I will use subprocess communication like in "good" old days...
I know how to call a command and to get its output:
out := LibC resultOfCommand: 'dir ', aDir.
but I am talking about another scenario: a communication with a running process interactively (for example, with SSH or similar like in the example above - python.exe).
PS. Maybe it's possible to do it with LibC #pipe:mode even?
Let me start with that the PipeableOsProcess is probably broken on Windows. I have tried it and it just opened a command line and nothing else (it does not freeze my Pharo 8). The whole OSProcess does not work correctly in my eyes.
So I took a shot at LibC which is supposed to not work with Windows.
I’m a module defining access to standard LibC. I’m available under Linux and OSX, but not under Windows for obvious reasons :)
Next is to say that Python's Windows support is probably much better than Pharo's.
The solution, which is more like a workaround using files, is to use LibC and #runCommand: (I tried to come up with a similar example as you had shown above):
| count command result outputFile errorFile |
count := 9+1. "The counting"
command := 'echo ', count asString. "command run at the command line"
outputFile := 'output'. "a file into which the output is redirected"
errorFile := 'error'. "a file where the error output is redirected "
result := LibC runCommand: command, "run the command "
' >', outputFile, "redirect the output to output file"
' 2>', errorFile.
"reading back the value from output file"
outputFile asFileReference contents lines.
"reading back the value from the error file - which is empty in this case"
errorFile asFileReference contents lines.