Xml Document
I am having a problem regarding the xml attribute names coming from sharepoint which contains the attributes names like description0,ows_x0020_long_desc coming in the xmldoc
<z:row ows_LinkFilename="Aerospace Energy.jpg"
ows_Title="Aerospace"
ows_ContentType="Image"
ows__ModerationStatus="0"
ows_PreviewOnForm="Aerospace Energy.jpg"
ows_ThumbnailOnForm="Technology Experience/Aerospace Energy.jpg"
ows_Modified="2011-12-07 12:02:34"
ows_Editor="1073741823;#System Account"
ows_Description0="Honeywell's SmartPath® Ground-Based Augmentation System (GBAS), which offers airports improved efficiency and capacity, greater navigational accuracy, and fewer weather-related delays."
ows_ID="28"
ows_Created="2011-12-02 11:26:01"
ows_Author="1073741823;#System Account"
ows_FileSizeDisplay="6091"
ows_Mode="Energy"
ows_Solution="Business"
ows_Long_x0020_Desc="Honeywell's SmartTraffic™ and IntuVue® 3-D Weather Radar technologies make the skies safer and enable pilots to more efficiently route flights. SmartTraffic ."
ows_Brief_x0020_Desc="Honeywell's Required Navigation Performance (RNP) capabilities enable aircraft to fly more precise approaches through tight corridors and congested airports, leading to fewer delays."
ows_Tags="True"
ows__Level="1"
ows_UniqueId="28;#{928FDA3E-94FA-47A5-A9AD-B5D98C12C18C}"
ows_FSObjType="28;#0"
ows_Created_x0020_Date="28;#2011-12-02 11:26:01"
ows_ProgId="28;#"
ows_FileRef="28;#Technology Experience/Aerospace Energy.jpg"
ows_DocIcon="jpg"
ows_MetaInfo="28;#Solution:SW|Business vti_thumbnailexists:BW|true vti_parserversion:SR|14.0.0.4762 Category:SW|Enter Choice #1 Description0:LW|Honeywell's SmartPath® Ground-Based Augmentation System (GBAS), which offers airports improved efficiency and capacity, greater navigational accuracy, and fewer weather-related delays. vti_stickycachedpluggableparserprops:VX|wic_XResolution Subject vti_lastheight vti_title vti_lastwidth wic_YResolution oisimg_imageparsedversion vti_lastwidth:IW|294 vti_author:SR|SHAREPOINT\\system vti_previewexists:BW|true vti_modifiedby:SR|SHAREPOINT\\system Long Desc:LW|Honeywell's SmartTraffic™ and IntuVue® 3-D Weather Radar technologies make the skies safer and enable pilots to more efficiently route flights. SmartTraffic . Keywords:LW| vti_foldersubfolderitemcount:IR|0 vti_lastheight:IW|172 ContentTypeId:SW|0x0101009148F5A04DDD49CBA7127AADA5FB792B00AADE34325A8B49CDA8BB4DB53328F21400623D4FCEEB2ADC4EA8269BF873F0BB6F _Author:SW| vti_title:SW|Aerospace wic_System_Copyright:SW| Mode:SW|Energy Tags:SW|True wic_YResolution:DW|96.0000000000000 oisimg_imageparsedversion:IW|4 Brief Desc:LW|Honeywell's Required Navigation Performance (RNP) capabilities enable aircraft to fly more precise approaches through tight corridors and congested airports, leading to fewer delays. _Comments:LW| wic_XResolution:DW|96.0000000000000 Subject:SW|Aerospace vti_folderitemcount:IR|0"
ows_Last_x0020_Modified="28;#2011-12-07 12:02:34"
ows_owshiddenversion="6"
ows_FileLeafRef="28;#Aerospace Energy.jpg"
ows_PermMask="0x7fffffffffffffff"
xmlns:z="#RowsetSchema" />
Could you please tell the solution for this.
SharePoint when returning data in xml will always use this fromat.
Field names will be prepended by ows_
Internal names of field will be used not display names.
Internal field names in SharePoint contain unicode equivalents for special characters
e.g. if you create a field with name 'Field Name' from SharePoint UI,
SharePoint will create internal name as 'Field_x0020_Name'
where 0020 is unicode representation of space.
If fields are created by code or feature however you can specify your own internal and display names.
So if you are parsing such xml you will have to code remembering these rules.
SharePoint does not add x0020 escape sequence in field's internal name unless there is a space in the display name while creating the field from UI.
Also once the field is created, changing the display name has no effect on the internal name of a field.
So if you create a field 'Long Desc' from UI and the later change the name to 'LongDesc', the internal name will still be Long_x0020_Desc.
Related
I am using cloud google vision API to extract text from Aadhaar and PAN. How can I get exact user details like name, father's name, and address?
Raw Data
ଭାରତ ସରକାର
Government of India
ଜିତ୍ୟାନନ୍ଦ ଖେମୁକୁ
NITYANANDA KHEMUDU
ପିତା : ସୀତାରାମ ଖେମୁକୁ
Father: Sitaram Khemudu
ଜନ୍ମ ତାରିଖ / DOB : 01.07.1999
ପୁରୁଷ / Male
ମୋ ଆଧାର, ମୋ ପରିଚୟ
I have built 5-6 OCR till date like aadhar, pan, ITR, Driving Linces etc., using google cloud vision API, I think you are looking for response like
{"pan_card_no":"ECXXXXXX123",
"name":"fshksj"
}
to get such response you need to built your own logic, here are some logic's i can share with you
Perform OCR on your document using Google_cloud_vision API and store that response into one array (Goggle gives logic line by line)
Like in above case if you want to grab DOB first you can build logic like i) if "DOB" in (list of item) then grab the numeric values
To get the name what you can do is dropping the unnecessary items from list by if using if condition like (if "India" in i) or (if i.isdigit()) then drop it likewise you can drop the unnesseary items from main list to get the Name
to grab the Address what you can do is, 95% of the time address come with pincode at last, so what you can do is treat pincode as a last index of address and look of "Address" kind of keyword then add all the elements from "Add keyword index" to "pincode index" ( this can be easily done in list) to validate whether the pincode is valid or not you can use library like Pyzipin
There are multiple conditions that you can use, above are the very basic one i mentioned, if you need any specific logic then then you can ask me
Imagine I have a report, a letter actually, which I need to translate to several languages. I have created a greeting field in the form which is filled programatically by an onchange event method.
if self.partner_id.gender == 'female':
self.letter_greeting = _('Dear %s %s,') % ( # the translation should be "Estimada"
self.repr_recipient_id.title.shorcut, surname
)
elif self.partner_id.gender == 'male':
self.letter_greeting = _('Dear %s %s,') % ( # translation "Estimado"
self.repr_recipient_id.title.shorcut, surname
)
else:
self.letter_greeting = _('Dear %s %s,') % ( # translation: "Estimado/a"
self.partner_id.title.shorcut, surname
)
In that case the word Dear should be translated to different Spanish translations depending on which option is used, this is because we use different termination depending on the gender. Exporting the po file I found that all the options are altogether, that make sense because almost all the cases the translations will be the same, but not in this case:
#. module: custom_module
#: code:addons/custom_module/models/sale_order.py:334
#: code:addons/custom_module/models/sale_order.py:338
#: code:addons/custom_module/models/sale_order.py:342
#, python-format
msgid "Dear %s %s,"
msgstr "Dear %s %s,"
Solutions I can apply directly
Put all the terms in different entries to avoid the same translation manually every time I need to update the po file. This can be cumbersome if you have many different words with that problem. If I do it and I open the file with poedit, this error appears: duplicate message definition
Put all the possible combinations with slashes, this is done y some other parts of Odoo. For both gender would be:
#. module: stock
#: model:res.company,msg:stock.res_company
msgid "Dear"
msgstr "Estimado/a"
This is just an example. I can think of many words that look the same in English, but they use different spelling or meanings in other languages depending on the context.
Possible best solutions
I don't know if Odoo know anything aboutu the context of a word to know if it was already translated or not. Adding a context manually could solve the problem, at least for words with different meanings.
The nicest solution would be to have a parameter to the translation module to make sure that the word is exported as an isolated entry for that especific translation.
Do you think that I am giving to it too much importance haha? Do you know if there is any better solution? Why is poedit not taking into account that problem at all?
I propose an extension of models res.partner.title and res.partner.
res.partner.title should get a translateable field for saving salutation prefixes like 'Dear' or 'Sehr geehrter' (German). Maybe it's worth to get something about genders, too, but i won't get into detail here about that.
You probably want to show the configuring user an example like "Dear Mr. Name" or something like that. A computed field should work.
On res.partner you should just implement either a computed field or just a method to get a full salutation for a partner record.
To some degree this is a linguistics problem. I believe the best solution would be to use a different "Source Language", one made up of keys, and then have English as another Translation. The word "Dear" in English does not have a gender context (and typically, much of English doesn't), while the word "Estimado" in Spanish does. The translation from that Spanish word to English is more appropriately "Masculine Dear." Therefore, using keys as your source language, you would have this:
SourceText (EnglishDescription) -> Translation (English) -> Translation (Spanish)
DearMasculine -> Dear -> Estimado
DearFeminine -> Dear -> Estimada
DearNuetral -> Dear -> Estimado/a
We have various rows inside our Elasticsearch index that contain the text
"... 2% milk ...".
User enters a query like "2% milk" into a search field and we transform it internally to a query
title:(+milk* +2%*)
because all terms should be required and we are possibly interested into rows that contain "2% milkfat".
This query above return zero hits. Changing the query to
title:(+milk* +2%)
returns reasonable results. So why does the '*' operator in the first query not work?
Unless you set a mapping, the "%" sign will get removed in the tokenization process. Basically "2% milk" will get turned into the tokens 2 and milk.
When you search for "2%*" it looks for tokens like: 2%, 2%a, 2%b, etc... and not match any indexed tokens, giving no hits.
When you search for "2%", it will go through the same tokenization process as at index-time (you can specify this, but the default tokenization is the same) and you will be looking for documents matching the token 2, which will give you a hit.
You can read more about the analysis/tokenization process here and you can set up the analysis you want by defining a custom mapping
Good luck!
Prefix and Wildcard queries do not appear to apply the Analyzer to their content. To provide a few examples:
title:(+milk* +2%) --> +title:milk* +title:2
title:(+milk* +2%*) --> +title:milk* +title:2%*
title:(+milk* +2%3) --> +title:milk* +(title:2 title:3)
title:(+milk* +2%3*) --> +title:milk* +title:2%3*
+title:super\\-milk --> +title:super title:milk
+title:super\\-milk* --> +title:super-milk*
It does make some sense to prevent tokenization of wildcard queries, since wildcard phrase queries are not allowed. If tokenization were allowed, it would seem to beg the question, especially with embeded wildcards, of just how many terms that wildcard can span.
Xml Document
I am having a problem regarding the xml attribute names coming from sharepoint which contains the attributes names like description0,ows_x0020_long_desc coming in the xmldoc
<z:row ows_LinkFilename="Aerospace Energy.jpg"
ows_Title="Aerospace"
ows_ContentType="Image"
ows__ModerationStatus="0"
ows_PreviewOnForm="Aerospace Energy.jpg"
ows_ThumbnailOnForm="Technology Experience/Aerospace Energy.jpg"
ows_Modified="2011-12-07 12:02:34"
ows_Editor="1073741823;#System Account"
ows_Description0="Honeywell's SmartPath® Ground-Based Augmentation System (GBAS), which offers airports improved efficiency and capacity, greater navigational accuracy, and fewer weather-related delays."
ows_ID="28"
ows_Created="2011-12-02 11:26:01"
ows_Author="1073741823;#System Account"
ows_FileSizeDisplay="6091"
ows_Mode="Energy"
ows_Solution="Business"
ows_Long_x0020_Desc="Honeywell's SmartTraffic™ and IntuVue® 3-D Weather Radar technologies make the skies safer and enable pilots to more efficiently route flights. SmartTraffic ."
ows_Brief_x0020_Desc="Honeywell's Required Navigation Performance (RNP) capabilities enable aircraft to fly more precise approaches through tight corridors and congested airports, leading to fewer delays."
ows_Tags="True"
ows__Level="1"
ows_UniqueId="28;#{928FDA3E-94FA-47A5-A9AD-B5D98C12C18C}"
ows_FSObjType="28;#0"
ows_Created_x0020_Date="28;#2011-12-02 11:26:01"
ows_ProgId="28;#"
ows_FileRef="28;#Technology Experience/Aerospace Energy.jpg"
ows_DocIcon="jpg"
ows_MetaInfo="28;#Solution:SW|Business vti_thumbnailexists:BW|true vti_parserversion:SR|14.0.0.4762 Category:SW|Enter Choice #1 Description0:LW|Honeywell's SmartPath® Ground-Based Augmentation System (GBAS), which offers airports improved efficiency and capacity, greater navigational accuracy, and fewer weather-related delays. vti_stickycachedpluggableparserprops:VX|wic_XResolution Subject vti_lastheight vti_title vti_lastwidth wic_YResolution oisimg_imageparsedversion vti_lastwidth:IW|294 vti_author:SR|SHAREPOINT\\system vti_previewexists:BW|true vti_modifiedby:SR|SHAREPOINT\\system Long Desc:LW|Honeywell's SmartTraffic™ and IntuVue® 3-D Weather Radar technologies make the skies safer and enable pilots to more efficiently route flights. SmartTraffic . Keywords:LW| vti_foldersubfolderitemcount:IR|0 vti_lastheight:IW|172 ContentTypeId:SW|0x0101009148F5A04DDD49CBA7127AADA5FB792B00AADE34325A8B49CDA8BB4DB53328F21400623D4FCEEB2ADC4EA8269BF873F0BB6F _Author:SW| vti_title:SW|Aerospace wic_System_Copyright:SW| Mode:SW|Energy Tags:SW|True wic_YResolution:DW|96.0000000000000 oisimg_imageparsedversion:IW|4 Brief Desc:LW|Honeywell's Required Navigation Performance (RNP) capabilities enable aircraft to fly more precise approaches through tight corridors and congested airports, leading to fewer delays. _Comments:LW| wic_XResolution:DW|96.0000000000000 Subject:SW|Aerospace vti_folderitemcount:IR|0"
ows_Last_x0020_Modified="28;#2011-12-07 12:02:34"
ows_owshiddenversion="6"
ows_FileLeafRef="28;#Aerospace Energy.jpg"
ows_PermMask="0x7fffffffffffffff"
xmlns:z="#RowsetSchema" />
Could you please tell the solution for this.
SharePoint when returning data in xml will always use this fromat.
Field names will be prepended by ows_
Internal names of field will be used not display names.
Internal field names in SharePoint contain unicode equivalents for special characters
e.g. if you create a field with name 'Field Name' from SharePoint UI,
SharePoint will create internal name as 'Field_x0020_Name'
where 0020 is unicode representation of space.
If fields are created by code or feature however you can specify your own internal and display names.
So if you are parsing such xml you will have to code remembering these rules.
SharePoint does not add x0020 escape sequence in field's internal name unless there is a space in the display name while creating the field from UI.
Also once the field is created, changing the display name has no effect on the internal name of a field.
So if you create a field 'Long Desc' from UI and the later change the name to 'LongDesc', the internal name will still be Long_x0020_Desc.
I need to write a text parser for the education domain which can extract out the information like institute, location, course etc from the free text.
Currently i am doing it through lucene, steps are as follows:
Index all the data related to institute, courses and location.
Making shingles of the free text and searching each shingle in location, course and institute index dir and then trying to find out which part of text represents location, course etc.
In this approach I am missing lot of cases like B.tech can be written as btech, b-tech or b.tech.
I want to know is there any thing available which can do all these kind of things, I have heard about Ling-pipe and Gate but don't know how efficient they are.
You definitely need GATE. GATE has 2 main most frequently used features (among thousands others): rules and dictionaries. Dictionaries (gazetteers in GATE's terms) allow you to put all possible cases like "B.tech", "btech" and so on in a single text file and let GATE find and mark them all. Rules (more precisely, JAPE-rules) allow you to define patterns in text. For example, here's pattern to catch MIT's postal address ("77 Massachusetts Ave., Building XX, Cambridge MA 02139"):
{Token.kind == number}(SP){Token.orth == uppercase}(SP){Lookup.majorType == avenue}(COMMA)(SP)
{Token.string == "Building"}(SP){Token.kind == number}(COMMA)(SP)
{Lookup.majorType == city}(SP){Lookup.majorType == USState}(SP){Token.kind == number}
where (SP) and (COMMA) - macros (just to make text shorter), {Somthing} - is annotation, , {Token.kind == number} - annotation "Token" with feature "kind" equal to "number" (i.e. just number in the text), {Lookup} - annotation that captures values from dictionary (BTW, GATE already has dictionaries for such things as US cities). This is quite simple example, but you should see how easily you can cover even very complicated cases.
I didn't use Lucene but in your case I would leave different forms of the same keyword as they are and just hold a link table or such. In this table I'd keep the relation of these different forms.
You may need to write a regular expression to cover each possible form of your vocabulary.
Be careful about your choice of analyzer / tokenizer, because words like B.tech can be easily split into 2 different words (i.e. B and tech).
You may want to check UIMA. As Lingpipe and Gate, this framework features text annotation, which is what you are trying to do. Here is a tutorial which will help you write an annotator for UIMA:
http://uima.apache.org/d/uimaj-2.3.1/tutorials_and_users_guides.html#ugr.tug.aae.developing_annotator_code
UIMA has addons, in particular one for Lucene integration.
You can try http://code.google.com/p/graph-expression/
example of Adress parsing rules
GraphRegExp.Matcher Token = match("Token");
GraphRegExp.Matcher Country = GraphUtils.regexp("^USA$", Token);
GraphRegExp.Matcher Number = GraphUtils.regexp("^\\d+$", Token);
GraphRegExp.Matcher StateLike = GraphUtils.regexp("^([A-Z]{2})$", Token);
GraphRegExp.Matcher Postoffice = seq(match("BoxPrefix"), Number);
GraphRegExp.Matcher Postcode =
mark("Postcode", seq(GraphUtils.regexp("^\\d{5}$", Token), opt(GraphUtils.regexp("^\\d{4}$", Token))))
;
//mark(String, Matcher) -- means creating chunk over sub matcher
GraphRegExp.Matcher streetAddress = mark("StreetAddress", seq(Number, times(Token, 2, 5).reluctant()));
//without new lines
streetAddress = regexpNot("\n", streetAddress);
GraphRegExp.Matcher City = mark("City", GraphUtils.regexp("^[A-Z]\\w+$", Token));
Chunker chunker = Chunkers.pipeline(
Chunkers.regexp("Token", "\\w+"),
Chunkers.regexp("BoxPrefix", "\\b(POB|PO BOX)\\b"),
new GraphExpChunker("Address",
seq(
opt(streetAddress),
opt(Postoffice),
City,
StateLike,
Postcode,
Country
)
).setDebugString(true)
);
B.tech can be written as btech, b-tech or b.tech
Lucene will let you do fuzzy searches based on the Levenshtein Distance. A query for roam~ (note the ~) will find terms like foam and roams.
That might allow you to match the different cases.