OTP Validation with Yubikey - issue with HMAC-SHA1 signature - one-time-password

I'm meeting an understanding issue with OTP validation for Yubikey
I'm trying to develop in flutter the Validation Protocol V2 (https://developers.yubico.com/OTP/Specifications/OTP_validation_protocol.html)
I don't understand how you obtain the signature
I used the vector test from https://developers.yubico.com/OTP/Specifications/Test_vectors.html
my code is
String apiKey = 'mG5be6ZJU1qBGz24yPh/ESM3UdU=';
String keyValue = 'id=1&nonce=jrFwbaYFhn0HoxZIsd9LQ6w2ceU&otp=vvungrrdhvtklknvrtvuvbbkeidikkvgglrvdgrfcdft';
var hmacSha1 = crypto.Hmac(crypto.sha1, apiKey.codeUnits);
crypto.Digest sha1Result = hmacSha1.convert(keyValue.codeUnits);
String hEncode64 = base64.encode(sha1Result.bytes);
final http.Response responseHttp = await http.get(
Uri.parse('https://api.yubico.com/wsapi/2.0/verify?' +
keyValue +
'&h=' +
hEncode64),
);
I don't find the same result as Vector Test (h=%2Bja8S3IjbX593/LAgTBixwPNGX4%3D). -> +ja8S3IjbX593/LAgTBixwPNGX4=
But my misunderstanding thing is : when I try to base64 decode your vector test, I can't because it is not utf8 format
and when i try with the test vector available on site, it doesn,'t work
https://api.yubico.com/wsapi/2.0/verify?id=1&otp=vvungrrdhvtklknvrtvuvbbkeidikkvgglrvdgrfcdft&nonce=jrFwbaYFhn0HoxZIsd9LQ6w2ceU&h=%2Bja8S3IjbX593/LAgTBixwPNGX4%3D
h=JE5WcMcXV7vooWkeN2/7A4DpMFo=
t=2021-12-15T12:51:37Z0635
status=BAD_SIGNATURE
Could you help me to understand please my pb

TL;DR: See this bash implementation.
I guess you finally managed to get it running with Dart, but the API can even be curled.
I'm posting this here for future reference for whoever encounters the BAD_SIGNATURE with yubikey in bash.
To correctly generate the HMAC SHA1 with openssl, I had to:
Take the secret key generated when registering a key for an app (i.e.: mG5be6ZJU1qBGz24yPh/ESM3UdU= in the OP's post) and decode it from base64. This generates raw binary data.
Convert the raw binary data to a hex string via od.
Convert the hex string to uppercase via sed.
Feed everything into openssl to generate the HMAC SHA1 result.
Reinterpret the openssl result as binary via xxd.
Reencode it to base64.
This finally generated the correct signature.
I've published the steps in the TL;DR in the beginning.
Here's a snapshot:
#!/bin/bash
echo -n "YubiKey token: "
read token
id=$(cat creds | head -1 | cut -d':' -f2)
secret_key=$(cat creds | tail -1 | cut -d':' -f2 | base64 -d | od -A n -v -t x1 | tr -d ' \n')
nonce=$(echo -n $RANDOM | sha256sum | head -c 40 | cut -d ' ' -f1)
challenge="id=$id&nonce=$nonce&otp=$token&sl=100&timeout=8"
signature=$(echo -n $challenge | openssl dgst -sha1 -mac HMAC -macopt hexkey:$(echo -n "$secret_key") | sed 's/^(stdin)= //' | sed -E 's/([[:lower:]])|([[:upper:]])/\U\1\U\2/g'| xxd -r -p | base64)
response=$(curl --silent "https://api.yubico.com/wsapi/2.0/verify?$challenge&h=$signature" | tr '\r' '\n')
echo $response
This uses a file called creds (obtained by registering an API key here), which contains the following:
Client ID:XXXXX
Secret key:XXXXXXXXXXXXXXXX

Related

Get full list of Groups and Projects in Gitlab Cloud

I'm trying to get a full list of Projects and groups in out Gitlab cloud account.
I'm currently using their documentation as reference (bear in mind though I'm no developer) and using Linux command line to do so. Here's the documentation I'm trying to use:
https://docs.gitlab.com/ee/api/projects.html
https://docs.gitlab.com/ee/api/groups.html#list-a-groups-projects
I'm using the following command to get the data and parse in a readable format that I will export to csv or spreadsheet afterwards:
curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/?owned=yes&per_page=1000&page=1" | python -m json.tool | grep -E "http_url_to_repo|visibility" | awk '!(NR%2){print$0p}{p=$0}' | awk '{print $4,$2}' | sed -E 's/\"|\,//g' > gitlab.txt
My problem is that the code only return about 100 of the 280 repositories we have. It doesn't seem to get it recursively from all the groups and subgroups.
Any ideas on how I can improve this search to get everything ?
Thank you
It seems it can get only 100 per page so you will have to run it two times - first with page=1 and next with page=2. And for second page you will need >> to append to existing file gitlab.txt
curl --header "..." "https://...&per_page=100&page=1" | ... > gitlab.txt
curl --header "..." "https://...&per_page=100&page=2" | ... >> gitlab.txt
Or you will have to write script which first get all pages and later send it to pipe. You may also try to use for-loop in bash

LDAPSEARCH into table format

Is there any way to perform a LDAP search and save the results into a table format (e.g. csv)?
Cheers
Jorge
You can use the excellent miller tool (mlr)
The last bit:
echo output | sed 's/://g' | mlr --x2c cat then unsparsify
How it works:
the sed converts the output to XTAB format
--x2c converts XTAB to CSV
cat then unsparsify makes sure the missing values are just filled instead of breaking to different csv output
Total command:
ldapsearch -H ldap://<hostname>:389 -D "<bindDN>" -W -b "<base>" '<query>' -oldif-wrap=no -LLL cn mail telephoneNumber | sed 's/://g' | mlr --x2c cat then unsparsify
Just in case someone else has to do this:
Based on the answer provided in
Filter ldapsearch with awk/bash
this will output the LDAP info into a csv format:
$ ldapsearch -x -D "cn=something" | awk -v OFS=',' '{split($0,a,": ")} /^mail:/{mail=a[2]} /^uidNumber:/{uidNumber=a[2]} /^uid:/{uid=a[2]} /^cn/{cn=a[2]; print uid, uidNumber,cn , mail}' > ldap_dump.csv
NOTE
You need to be careful about the order in which you parse the LDAP data with awk! It needs to be parsed in the same order as it appears on the LDAP data!

Convert a column value in SHA256 in bash

I have a big csv file delimited with "|". How do I encrypt the 2nd column value into SHA256 using bash for full file.
Thanks in advance
for data in cat test.txt; do d1=$(echo $data | cut -d "|" -f1); d2=$(echo $data | cut -d "|" -f2 | sha256sum); rest=$(echo $data | cut -d "|" -f3-); echo "$d1|$d2|$rest";done >> new_file.txt

How to extract table data from PDF as CSV from the command line?

I want to extract all rows from here while ignoring the column headers as well as all page headers, i.e. Supported Devices.
pdftotext -layout DAC06E7D1302B790429AF6E84696FCFAB20B.pdf - \
| sed '$d' \
| sed -r 's/ +/,/g; s/ //g' \
> output.csv
The resulting file should be in CSV spreadsheet format (comma separated value fields).
In other words, I want to improve the above command so that the output doesn't brake at all. Any ideas?
I'll offer you another solution as well.
While in this case the pdftotext method works with reasonable effort, there may be cases where not each page has the same column widths (as your rather benign PDF shows).
Here the not-so-well-known, but pretty cool Free and OpenSource Software Tabula-Extractor is the best choice.
I myself am using the direct GitHub checkout:
$ cd $HOME ; mkdir svn-stuff ; cd svn-stuff
$ git clone https://github.com/tabulapdf/tabula-extractor.git git.tabula-extractor
I wrote myself a pretty simple wrapper script like this:
$ cat ~/bin/tabulaextr
#!/bin/bash
cd ${HOME}/svn-stuff/git.tabula-extractor/bin
./tabula $#
Since ~/bin/ is in my $PATH, I just run
$ tabulaextr --pages all \
$(pwd)/DAC06E7D1302B790429AF6E84696FCFAB20B.pdf \
| tee my.csv
to extract all the tables from all pages and convert them to a single CSV file.
The first ten (out of a total of 8727) lines of the CVS look like this:
$ head DAC06E7D1302B790429AF6E84696FCFAB20B.csv
Retail Branding,Marketing Name,Device,Model
"","",AD681H,Smartfren Andromax AD681H
"","",FJL21,FJL21
"","",Luno,Luno
"","",T31,Panasonic T31
"","",hws7721g,MediaPad 7 Youth 2
3Q,OC1020A,OC1020A,OC1020A
7Eleven,IN265,IN265,IN265
A.O.I. ELECTRONICS FACTORY,A.O.I.,TR10CS1_11,TR10CS1
AG Mobile,Status,Status,Status
which in the original PDF look like this:
It even got these lines on the last page, 293, right:
nabi,"nabi Big Tab HD\xe2\x84\xa2 20""",DMTAB-NV20A,DMTAB-NV20A
nabi,"nabi Big Tab HD\xe2\x84\xa2 24""",DMTAB-NV24A,DMTAB-NV24A
which look on the PDF page like this:
TabulaPDF and Tabula-Extractor are really, really cool for jobs like this!
Update
Here is an ASCiinema screencast (which you also can download and re-play locally in your Linux/MacOSX/Unix terminal with the help of the asciinema command line tool), starring tabula-extractor:
As Martin R commented, tabula-java is the new version of tabula-extractor and active. 1.0.0 was released on July 21st, 2017.
Download the jar file and with the latest java:
java -jar ./tabula-1.0.0-jar-with-dependencies.jar \
--pages=all \
./DAC06E7D1302B790429AF6E84696FCFAB20B.pdf
> support_devices.csv
What you want is rather easy, but you're having a different problem also (I'm not sure you are aware of it...).
First, you should add -nopgbrk for ("No pagebreaks, please!") to your command. Because these pesky ^L characters which otherwise appear in the output then need not be filtered out later.
Adding a grep -vE '(Supported Devices|^$)' will then filter out all the lines you do not want, including empty lines, or lines with only spaces:
pdftotext -layout -nopgbrk \
DAC06E7D1302B790429AF6E84696FCFAB20B.pdf - \
| grep -vE '(Supported Devices|^$|Marketing Name)' \
| gsed '$d' \
| gsed -r 's# +#,#g' \
| gsed '# ##g' \
> output2.csv
However, your other problem is this:
Some of the table fields are empty.
Empty fields appear with the -layout option as a series of space characters, sometimes even two in the same row.
However, the text columns are not spaced identically from page to page.
Therefor you will not know from line to line how many spaces you need to regard as a an "empty CSV field" (where you'd need an extra , separator).
As a consequence, your current code will show only one, two or three (instead of four) fields for some lines, and these fields end up in the wrong columns!
There is a workaround for this:
Add the -x ... -y ... -W ... -H ... parameters to pdftotext to crop the PDF column-wise.
Then append the columns with a combination of utilities like paste and column.
The following command extracts the first columns:
pdftotext -layout -x 38 -y 77 -W 176 -H 500 \
DAC06E7D1302B790429AF6E84696FCFAB20B.pdf - > 1st-columns.txt
These are for second, third and fourth columns:
pdftotext -layout -x 214 -y 77 -W 176 -H 500 \
DAC06E7D1302B790429AF6E84696FCFAB20B.pdf - > 2nd-columns.txt
pdftotext -layout -x 390 -y 77 -W 176 -H 500 \
DAC06E7D1302B790429AF6E84696FCFAB20B.pdf - > 3rd-columns.txt
pdftotext -layout -x 567 -y 77 -W 176 -H 500 \
DAC06E7D1302B790429AF6E84696FCFAB20B.pdf - > 4th-columns.txt
BTW, I cheated a bit: in order to get a clue about what values to use for -x, -y, -W and -H I did first run this command in order to find the exact coordinates of the column header words:
pdftotext -f 1 -l 1 -layout -bbox \
DAC06E7D1302B790429AF6E84696FCFAB20B.pdf - | head -n 10
It's always good if you know how to read and make use of pdftotext -h. :-)
Anyway, how to append the four text files as columns side by side, with the proper CVS separator in between, you should find out yourself. Or ask a new question :-)
This can be done easily with an IntelliGet (http://akribiatech.com/intelliget) script as below
userVariables = brand, name, device, model;
{ start = Not(Or(Or(IsSubstring("Supported Devices",Line(0)),
IsSubstring("Retail Branding",Line(0))),
IsEqual(Length(Trim(Line(0))),0)));
brand = Trim(Substring(Line(0),10,44));
name = Trim(Substring(Line(0),45,79));
device = Trim(Substring(Line(0),80,114));
model = Trim(Substring(Line(0),115,200));
output = Concat(brand, ",", name, ",", device, ",", model);
}
For the case where you want to extract that tabular data from PDF over which you have control at creation time (for timesheets contracts your employees have to sign), the following solution will be cleaner:
Create a PDF form with field IDs.
Let people fill and save the PDF forms.
Use a Apache PDFBox, an open source tool that allows to extract form data from a PDF. It includes a command-line example tool PrintFields that you would call as follows to print the desired field information:
org.apache.pdfbox.examples.interactive.form.PrintFields file.pdf
For other options, see this question.
As an alternative to the above workflow, maybe you could also use a digital signature web service that allows PDF form filling and export of the data to tables. Such as SignRequest, which allows to create templates and later export the data of signed documents. (Not affiliated, just found this myself.)

redis-cli and value from a file

Is it possible to easily set specific value from a file using interactive redis-cli?
I'd like to achieve same result as with following Python snippet:
with open("some.jpg") as f:
image_binary = f.read()
rd.hset("some_key", "image_binary", image_binary)
Is it possible to easily set specific value from a file using interactive redis-cli?
Since -x reads the last argument from STDIN, what about:
redis-cli -x HSET some_key image_binary <some.jpg
You can then easily retrieve the file as follow:
redis-cli --raw HGET some_key image_binary > img.jpg
Note that there is an extra \n character at the end.
A different approach is to feed redis-cli a sequence of commands written in a text file:
$ cat /tmp/commands.txt
SET item:3374 100
INCR item:3374
APPEND item:3374 xxx
GET item:3374
$ cat /tmp/commands.txt | redis-cli
OK
(integer) 101
(integer) 6
"101xxx"
Taken from : Redis Cli Manual