I want to ask that i have a column of claims in which a claim no have different occurances but i want to assign value of 1 when claim no is distinct and it occurred first time then i assign 0 when it is repeated , so is there any method for creating calculated column for this problem
You can use Power Query to achieve your goal: Below I created some datasets, and achieved a result:
Let's say you have such data set:
Then here is all steps involved in Power Query to achieve your goal. Just Paste the code into Advanced Editor in which you can find in Home tab and Query Group:
If you clicked the Advanced Editor, You can see the full code, like this:
The Full M-Code:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjMwNDFS0lEyVIrVAfNMzYA8IxgPTc4UWQ6s0hBFpSmKnAmKnBGKKcboZsYCAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Claim No" = _t, Occurances = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Claim No", Int64.Type}, {"Occurances", Int64.Type}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Claim No"}, {{"Order", each _, type table [Claim No=nullable number, Occurances=nullable number]}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom", each Table.AddIndexColumn([Order],"SNo",1)),
Custom1 = Table.Combine(#"Added Custom"[Custom]),
#"Added Custom1" = Table.AddColumn(Custom1, "Distinct_Number", each if [SNo] = 1 then 1 else 0),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom1",{"SNo"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Removed Columns",{{"Distinct_Number", Int64.Type}})
in
#"Changed Type1"
If we test it, It returns:
I need to expand a table column but some of the table contents are empty and return ""
when I run the query it stops at the first table where the content is empty.
I need to ignore these rows when using the Table.ExpandTableColumn or delete them beforehand.
#"Função Personalizada Invocada" = Table.AddColumn(#"Tipo Alterado", "generation_24hrs", each get_generation_24([user.data.id])),
#"generation_24hrs Expandido" = Table.ExpandTableColumn(#"Função Personalizada Invocada", "generation_24hrs", {"entry.value.id", "entry.value.kwh_ger", "entry.value.kwh_con"}, {"entry.value.id", "entry.value.kwh_ger", "entry.value.kwh_con"})
Before
After:
try
#"Função Personalizada Invocada" = Table.AddColumn(#"Tipo Alterado", "generation_24hrs", each get_generation_24([user.data.id])),
#"Added Custom" = Table.AddColumn(#"Função Personalizada Invocada", "RowCount", each Table.RowCount([generation_24hrs])),
#"Filtered Rows" = Table.SelectRows(#"Added Custom", each ([RowCount] <> "0")),
#"generation_24hrs Expandido" = Table.ExpandTableColumn(#"Filtered Rows", "generation_24hrs", {"entry.value.id", "entry.value.kwh_ger", "entry.value.kwh_con"}, {"entry.value.id", "entry.value.kwh_ger", "entry.value.kwh_con"})
I prepared a scientometric dashboard using Power BI which is connected directly to Scopus database by calling API keys. On my Power Bi desktop it can refresh data without any error but after publishing the dashboard to power bi website it can not be refreshed and returns a credential error:
Last refresh failed: Wed Nov 27 2019 12:32:39 GMT+0330 (Iran Standard
Time) There was an error when processing the data in the dataset.Hide
details Message: The credentials provided for the Web source are
invalid. (Source at https://api.elsevier.com/content/search/scopus.)
Table: API-Scopus-All. Cluster
URI: WABI-EAST-ASIA-A-PRIMARY-redirect.analysis.windows.net Activity
ID: 7edc8fb9-5513-465d-a35b-70cc5629d0d0 Request
ID: 2edb255e-20fe-d1db-6b7d-2cf1b6681fc5 Time: 2019-11-27 09:02:39Z
following code is my query in Power BI. Moreover my credential on desktop is "Basic" with "User name"= my apikey
I only deleted my apikey from code. Anyone wants to reproduce results, should replace his/her Scopus apikey with APIKEY. Moreover access to Scopus database should be provided.
I appreciate any help for solving the credential issue. thanks
let
Source = 1000, //the total value from a total rows api?
Starts = List.Generate(()=>0, each _ < Source, each _ + 25),
#"Converted to Table" = Table.FromList(Starts, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Changed Type" = Table.TransformColumnTypes(#"Converted to Table",{{"Column1", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Custom", each Json.Document(Web.Contents(
"https://api.elsevier.com/",
[
RelativePath="content/search/scopus/",
Query=
[
view="complete",
count="25",
query="AFFIL ( {Environmental Research Center} OR {Institute for Environmental Research} ) AND AFFIL ( {Tehran University of Medical Sciences} OR {Tehran University of Medical Science} ) AND AFFIL ( {Netherlands})",
apiKey="APIKEY",
limit="40",
start=""&[Column1]
]
]
))),
#"Expanded Custom" = Table.ExpandRecordColumn(#"Added Custom", "Custom", {"search-results"}, {"Custom.search-results"}),
#"Expanded Custom.search-results" = Table.ExpandRecordColumn(#"Expanded Custom", "Custom.search-results", {"opensearch:totalResults", "opensearch:startIndex", "opensearch:itemsPerPage", "opensearch:Query", "link", "entry"}, {"Custom.search-results.opensearch:totalResults", "Custom.search-results.opensearch:startIndex", "Custom.search-results.opensearch:itemsPerPage", "Custom.search-results.opensearch:Query", "Custom.search-results.link", "Custom.search-results.entry"}),
#"Expanded Custom.search-results.opensearch:Query" = Table.ExpandRecordColumn(#"Expanded Custom.search-results", "Custom.search-results.opensearch:Query", {"#role", "#searchTerms", "#startPage"}, {"Custom.search-results.opensearch:Query.#role", "Custom.search-results.opensearch:Query.#searchTerms", "Custom.search-results.opensearch:Query.#startPage"}),
#"Expanded Custom.search-results.link" = Table.ExpandListColumn(#"Expanded Custom.search-results.opensearch:Query", "Custom.search-results.link"),
#"Expanded Custom.search-results.link1" = Table.ExpandRecordColumn(#"Expanded Custom.search-results.link", "Custom.search-results.link", {"#_fa", "#ref", "#href", "#type"}, {"Custom.search-results.link.#_fa", "Custom.search-results.link.#ref", "Custom.search-results.link.#href", "Custom.search-results.link.#type"}),
#"Expanded Custom.search-results.entry" = Table.ExpandListColumn(#"Expanded Custom.search-results.link1", "Custom.search-results.entry"),
#"Expanded Custom.search-results.entry1" = Table.ExpandRecordColumn(#"Expanded Custom.search-results.entry", "Custom.search-results.entry", {"#_fa", "link", "prism:url", "dc:identifier", "eid", "dc:title", "dc:creator", "prism:publicationName", "prism:issn", "prism:eIssn", "prism:volume", "prism:pageRange", "prism:coverDate", "prism:coverDisplayDate", "prism:doi", "pii", "dc:description", "citedby-count", "affiliation", "prism:aggregationType", "subtype", "subtypeDescription", "author-count", "author", "authkeywords", "article-number", "source-id", "fund-acr", "fund-no", "fund-sponsor", "openaccess", "openaccessFlag"}, {"Custom.search-results.entry.#_fa", "Custom.search-results.entry.link", "Custom.search-results.entry.prism:url", "Custom.search-results.entry.dc:identifier", "Custom.search-results.entry.eid", "Custom.search-results.entry.dc:title", "Custom.search-results.entry.dc:creator", "Custom.search-results.entry.prism:publicationName", "Custom.search-results.entry.prism:issn", "Custom.search-results.entry.prism:eIssn", "Custom.search-results.entry.prism:volume", "Custom.search-results.entry.prism:pageRange", "Custom.search-results.entry.prism:coverDate", "Custom.search-results.entry.prism:coverDisplayDate", "Custom.search-results.entry.prism:doi", "Custom.search-results.entry.pii", "Custom.search-results.entry.dc:description", "Custom.search-results.entry.citedby-count", "Custom.search-results.entry.affiliation", "Custom.search-results.entry.prism:aggregationType", "Custom.search-results.entry.subtype", "Custom.search-results.entry.subtypeDescription", "Custom.search-results.entry.author-count", "Custom.search-results.entry.author", "Custom.search-results.entry.authkeywords", "Custom.search-results.entry.article-number", "Custom.search-results.entry.source-id", "Custom.search-results.entry.fund-acr", "Custom.search-results.entry.fund-no", "Custom.search-results.entry.fund-sponsor", "Custom.search-results.entry.openaccess", "Custom.search-results.entry.openaccessFlag"}),
#"Expanded Custom.search-results.entry.link" = Table.ExpandListColumn(#"Expanded Custom.search-results.entry1", "Custom.search-results.entry.link"),
#"Expanded Custom.search-results.entry.link1" = Table.ExpandRecordColumn(#"Expanded Custom.search-results.entry.link", "Custom.search-results.entry.link", {"#_fa", "#ref", "#href"}, {"Custom.search-results.entry.link.#_fa", "Custom.search-results.entry.link.#ref", "Custom.search-results.entry.link.#href"}),
#"Expanded Custom.search-results.entry.affiliation" = Table.ExpandListColumn(#"Expanded Custom.search-results.entry.link1", "Custom.search-results.entry.affiliation"),
#"Expanded Custom.search-results.entry.affiliation1" = Table.ExpandRecordColumn(#"Expanded Custom.search-results.entry.affiliation", "Custom.search-results.entry.affiliation", {"#_fa", "affiliation-url", "afid", "affilname", "affiliation-city", "affiliation-country"}, {"Custom.search-results.entry.affiliation.#_fa", "Custom.search-results.entry.affiliation.affiliation-url", "Custom.search-results.entry.affiliation.afid", "Custom.search-results.entry.affiliation.affilname", "Custom.search-results.entry.affiliation.affiliation-city", "Custom.search-results.entry.affiliation.affiliation-country"}),
#"Expanded Custom.search-results.entry.author-count" = Table.ExpandRecordColumn(#"Expanded Custom.search-results.entry.affiliation1", "Custom.search-results.entry.author-count", {"#limit", "#total", "$"}, {"Custom.search-results.entry.author-count.#limit", "Custom.search-results.entry.author-count.#total", "Custom.search-results.entry.author-count.$"}),
#"Expanded Custom.search-results.entry.author" = Table.ExpandListColumn(#"Expanded Custom.search-results.entry.author-count", "Custom.search-results.entry.author"),
#"Expanded Custom.search-results.entry.author1" = Table.ExpandRecordColumn(#"Expanded Custom.search-results.entry.author", "Custom.search-results.entry.author", {"#_fa", "#seq", "author-url", "authid", "authname", "surname", "given-name", "initials", "afid"}, {"Custom.search-results.entry.author.#_fa", "Custom.search-results.entry.author.#seq", "Custom.search-results.entry.author.author-url", "Custom.search-results.entry.author.authid", "Custom.search-results.entry.author.authname", "Custom.search-results.entry.author.surname", "Custom.search-results.entry.author.given-name", "Custom.search-results.entry.author.initials", "Custom.search-results.entry.author.afid"}),
#"Expanded Custom.search-results.entry.author.afid" = Table.ExpandListColumn(#"Expanded Custom.search-results.entry.author1", "Custom.search-results.entry.author.afid"),
#"Expanded Custom.search-results.entry.author.afid1" = Table.ExpandRecordColumn(#"Expanded Custom.search-results.entry.author.afid", "Custom.search-results.entry.author.afid", {"#_fa", "$"}, {"Custom.search-results.entry.author.afid.#_fa", "Custom.search-results.entry.author.afid.$"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Custom.search-results.entry.author.afid1",{"Column1", "Custom.search-results.opensearch:startIndex", "Custom.search-results.opensearch:itemsPerPage", "Custom.search-results.opensearch:Query.#role", "Custom.search-results.opensearch:Query.#searchTerms", "Custom.search-results.opensearch:Query.#startPage", "Custom.search-results.link.#_fa", "Custom.search-results.link.#type", "Custom.search-results.entry.#_fa", "Custom.search-results.entry.link.#_fa", "Custom.search-results.entry.link.#ref", "Custom.search-results.entry.link.#href", "Custom.search-results.entry.prism:issn", "Custom.search-results.entry.prism:eIssn", "Custom.search-results.entry.prism:volume", "Custom.search-results.entry.prism:pageRange", "Custom.search-results.entry.dc:description", "Custom.search-results.entry.affiliation.#_fa", "Custom.search-results.entry.author-count.#limit", "Custom.search-results.entry.author.#_fa", "Custom.search-results.entry.author.afid.#_fa", "Custom.search-results.entry.article-number", "Custom.search-results.entry.source-id", "Custom.search-results.link.#href"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Removed Columns",{{"Custom.search-results.entry.citedby-count", Int64.Type}}),
#"Renamed Columns" = Table.RenameColumns(#"Changed Type1",{{"Custom.search-results.entry.prism:doi", "DOI"}}),
#"Added Custom1" = Table.AddColumn(#"Renamed Columns", "URL", each "https://doi.org/"&[DOI]),
#"Duplicated Column" = Table.DuplicateColumn(#"Added Custom1", "Custom.search-results.entry.prism:coverDate", "Custom.search-results.entry.prism:coverDate - Copy"),
#"Renamed Columns1" = Table.RenameColumns(#"Duplicated Column",{{"Custom.search-results.entry.prism:coverDate - Copy", "Date"}}),
#"Changed Type2" = Table.TransformColumnTypes(#"Renamed Columns1",{{"Date", type date}}),
#"Renamed Columns2" = Table.RenameColumns(#"Changed Type2",{{"Custom.search-results.entry.prism:coverDate", "Cover date"}}),
#"Changed Type3" = Table.TransformColumnTypes(#"Renamed Columns2",{{"Cover date", type date}})
in
#"Changed Type3"
The problem is not related to IP client:
because if I make my query in Scopus:
(https://dev.elsevier.com/search.html#!/Scopus_Search/ScopusSearch) to produce a URL:
(https://api.elsevier.com/content/search/scopus?query=AFFIL%20(%20%7BEnvironmental%20Research%20Center%7D%20%20OR%20%20%7BInstitute%20for%20Environmental%20Research%7D%20)%20%20AND%20%20AFFIL%20(%20%7BTehran%20University%20of%20Medical%20Sciences%7D%20%20OR%20%20%7BTehran%20University%20of%20Medical%20Science%7D%20)%20AND%20%20AFFIL%20(%20%7BNetherlands%7D)&apiKey="MY-API-KEY")
and import it as web address to PBI it works well and can be refreshed without problem after publishing to PBI web. But the problem is with this simple query only one page of Scopus searched items will be returned. I need all, and because of it I changed the code as above, but it can not be refreshed after publish to PBI!
The issue is probably related to the fact that by default, the Scopus API uses the client's IP address to check if it has a subscription to Scopus. When you run the dashboard on your Power BI desktop client, requests to the Scopus API are sent from your client's IP address, and if your client IP address is set up for access to Scopus, you will get full access to Scopus data through the API as well. But when you run your dashboard on the Power BI website, requests to the Scopus API are probably sent from the Power BI server's IP address, which may not be set up for access to Scopus. Depending on your use case, you may be able to request the use of an authentication token by contacting Scopus API support.
I am stuck at the error handling routine. I have this function..
(LicenceNumber) =>
let
Source =
Web.Page(Web.Contents("http://mbsweblist.fsco.gov.on.ca/ShowLicence.aspx?M"
& Number.ToText(LicenceNumber) & "~")),
WebData = Source{1}[Data],
#"Extracted Text Before Delimiter" = Table.TransformColumns(WebData,
{{"Column1", each Text.BeforeDelimiter(_, ":"), type text}}),
#"Removed Top Rows" = Table.Skip(#"Extracted Text Before Delimiter",1),
#"Transposed Table" = Table.Transpose(#"Removed Top Rows"),
#"Promoted Headers" = Table.PromoteHeaders(#"Transposed Table",
[PromoteAllScalars=true])
in
#"Promoted Headers"
Which returns data to table
let
Source = {13000246..13000250},
#"Convert to Table" = Table.FromList(Source,Splitter.SplitByNothing(),{"Licence Number"}),
#"Changed Type" = Table.TransformColumnTypes(#"Convert to Table",{{"Licence Number", Int64.Type}}),
#"Get WebData" = Table.AddColumn(#"Changed Type", "WebData", each try WebData([Licence Number]) otherwise #table({},{})),
#"Combine WebData" = Table.Combine(#"Get WebData"[WebData]),
#"Changed Types" = Table.TransformColumnTypes(#"Combine WebData",{{"Agent/Broker Name", type text}, {"Licence #", type text}, {"Brokerage Name", type text}, {"Licence Class", type text}, {"Status", type text}, {"Issue Date", type date}, {"Expiry Date", type date}, {"Inactive Date", type date}})
in
#"Changed Types"
I am trying to error handle a situation where I pass an invalid value in source, lets say source = {13009995..13009999}, this is throwing error - "col X of table was not found". I tried to use the following error handling logic but it is not working ..
Empty = #table({{"Agent/Broker Name", type text}, {"Licence #", type text},
{"Brokerage Name", type text}, {"Licence Class", type text}, {"Status", type
text}, {"Issue Date", type date}, {"Expiry Date", type date}, {"Inactive
Date", type date}},{}),
Combine = Table.Combine({#"Get WebData"[WebData], Empty}),
I am primarily a business analyst and unable to fix this error. Requesting help. User Olly had helped me with my primary query
I would suggest creating an empty table as a separate query called EmptyTable that matches the columns when you do get data back. Here's the M code for that:
let
Empty =
#table(
{
"Agent/Broker Name",
"Licence #",
"Brokerage Name",
"Licence Class",
"Status",
"Issue Date",
"Expiry Date",
"Inactive Date"
},
{}
)
in
Empty
Now in your #"Get WebData" step, simply swap out #table({},{}) for EmptyTable.
#"Get WebData" =
Table.AddColumn(
#"Changed Type",
"WebData",
each
try WebData([Licence Number])
otherwise EmptyTable
),
Note: Your query looks to work fine when there is at least one valid license number.
Using PostgreSQL, I would like to be able to only see Document IDs with the latest modification time stamp. I am having difficulty getting this working and was wondering if anyone had any pointers?
Here is my current code:
SELECT cmsdw_document.document_id as "Document ID",
cmsdw_activity_meta.activity_name as "Activity Name",
cmsdw_document.title as "Title",
cmsdw_document.creation_ts as "Creation Timestamp",
cmsdw_document.modification_ts as "Modification Timestamp",
cmsdw_user.firstname as "First Name",
cmsdw_user.lastname as "Last Name",
cmsdw_container.name as "Name",
cmsdw_document_stats_fact.content_id as "Content ID",
cmsdw_document_stats_fact.views as "Views",
cmsdw_document_stats_fact.likes as "Likes",
cmsdw_document_stats_fact.bookmarks as "Bookmarks",
cmsdw_document_stats_fact.comments as "Comments",
cmsdw_document_stats_fact.shares as "Shares",
cmsdw_document_stats_fact.unique_viewers as "Unique Viewers"
FROM
public.cmsdw_document,
public.cmsdw_document_stats_fact,
public.cmsdw_container,
public.cmsdw_object,
public.cmsdw_user,
public.cmsdw_activity_fact,
public.cmsdw_activity_meta
WHERE
cmsdw_activity_fact.activity_type = cmsdw_activity_meta.activity_type AND
cmsdw_document_stats_fact.content_id = cmsdw_object.object_id AND
cmsdw_document.document_id = cmsdw_object.object_id AND
cmsdw_container.container_id = cmsdw_document.container_id AND
cmsdw_object.dw_object_id = cmsdw_activity_fact.direct_dw_object_id AND
cmsdw_object.object_type = cmsdw_activity_fact.direct_object_type AND
cmsdw_activity_fact.user_id = cmsdw_user.user_id AND
cmsdw_container.name = 'Getting Started' AND
cmsdw_object.object_type = 102 AND
cmsdw_activity_fact.activity_type = 20;
You should fix your query to have proper join syntax -- simple rule: never use commas in the from clause.
For your query, you can replace the select with with:
select distinct on (cmsdw_document.document_id) . . .
The ". . ." is the rest of your query. Then add:
order by cmsdw_document.document_id, cmsdw_document.modification_ts desc
This should give you the latest document, using a Postgres extension.