I have created a powerBI using Direct query to get live reports. But when changing slicer filter in visual it takes long time to load data in visual table. So I searched for alternate to direct query and found Push dataset. But on analysis, seems push dataset uses API for streaming. Is it possible to use the push dataset for the below select query(Have 15 columns and 20k rows) as simple as direct query.
EX: select * from persons p
left join students s on s.id=p.id
Related
I am using Excel's Power Query in order to test a SQL query that I am eventually going to use in order to make a pivot table that stays updated with the database. The database is accessed through an ODBC.
The problem is not related to Power Query itself but simply the SQL request.
Here I am trying to select all bills from the "facturation" (French database) table that are from the current year (2021). I am naming this selected data FACTURES_ANEE_COURANTE.
Then I want to also select some attributes of those items from 2021 in order to display them in the pivot table, but only on the selection that I just made in order to only select (and show) bills from the current year.
select * as FACTURES_ANNEE_COURANTE
from facturation
where year(date_fact)=2021 limit 3, select date_fact from FACTURES_ANNEE_COURANTE
I only have very basic knowledge of SQL and therefore this does not seem to work, the second part of my request that is (the first one works). I'm trying to do this in order to be able to show these specific attributes in the pivot table. What's the proper way to select attributes only from my first selection of elements from my table facturation?
Thank you for your help.
A major advantage of Power Query is being able to generate complex logic without needing to be able to code in SQL. So I would abandon writing hand coded SQL - there's no need.
Before PQ came out I had 2 decades of experience writing complex SQL. After PQ came out I've written almost none - the SQL code generated by PQ is good enough, you can easily add complex transformations that are hard/impossible in SQL, and overall developing and debugging is 10x easier.
For your scenario, I would build a PQ query just using the navigation to select your facturation table. Then I would use the PQ UI to Filter (instead of a SQL where clause) and Choose Columns (to restrict the columns returned).
Whatever other transformations you need are likely met using a button in the PQ UI.
I'm new to MS Access and am trying to speed up a data gathering process that is taking forever in Powershell. In Powershell I have 10 or so web API calls to get data and each comes back as an object with multiple properties (fields.) Each set of data has related fields to 1 or more of the other sets of data. Getting the data is very quick but piping an array of objects to where-object to select-object takes over an hour and there's really not that much data. Each object contains 500-1500 "records" and 5 to 10 "fields" so I thought why not export that data and use something that's intended to search through data to do the job? I exported each object as a separate .CSV file. So enter MS Access..
I imported each of the CSV's as a separate table (easy enough.) I'm going to simplify this down for this example to the following 3 tables:
[Tables]https://i.stack.imgur.com/UCH1F.jpg
Every table has fields that relate it over to other tables. Pretty much there's some sort of Id field in every table that is related to another Id field in a different table that I need to pull a field called "name" from. I'm trying to follow the bread crumbs from the Player name to it's Network name to it's Application name, to it's Layout name, etc... I want to build a query that I would eventually just be able to export as an Excel file. I also would prefer to just write out the SQL unless it's really easier to to understand the visual query builder. I'm looking to build a sheet with the following information:
Player's Name would include all names from the Players table and getting just that data makes sense to me. SELECT Name AS PlayerName FROM Players Everything else, not so much. I feel like this will end up being some mega query as I get deeper into related table after related table. In Excel, it would be straightforward using Vlookups across tabs but that doesn't seem to be the best approach. Given the info above, I'm trying to achieve the following output:
Result table
Any help with strategy and syntax greatly appreciated!
You're looking for the JOIN clause.
SELECT
Players.Name PlayerName, Networks.Name PlayerNetwork, Applications.Name ApplicationName
FROM
Players
LEFT OUTER JOIN
Networks
ON
Networks.ID = Players.NetworkId
LEFT OUTER JOIN
Applications
ON
Applications.Id = Players.ApplicationID
I need to run an ETL to get data from a Sybase/SQL back-end into my MongoDB environment. We already have data from Sybase, but now there are a couple of additional fields we want to pull data in for. So with my familiarity being with Mongo (not so much Sybase), I'm trying to determine how I need to adjust our ETL to get this additional data.
The current SELECT statement looks like this:
`SELECT DISTINCT TOP 100 d.*, d10.code code10, d10.id_number as Code10ID FROM diagnosis d LEFT JOIN diagnosis_icd10 d10 on d.icd10_id = d10.id_number ORDER BY d.id_number`
Now, within the diagnosis_icd10 table that we're doing the LEFT JOIN on, there are now a couple of extra fields available.
So, my question is, do I need to explicitly include these additional fields here in the SELECT statement in order for them to be available in the ETL process? Or is this only the case if I want to rename the fields? What should this look like?
Yes you need to explicitly request them, as your current query only gets all columns from the diagnosis table (aliased to d via the d*).
This has nothing to do with Sybase though this is basic SQL so would be the same for most databases. You just add them to the select statement as in d10.column_name like the others.
Been rattling my brain for a while and I could not get pass how to do the SQL query that will show the relationship/connections between my two tables.
I'm working on an IT equipment inventory program. I have two tables;
SELECT serial_number, model, ship_dat, status FROM items_list
SELECT item_serial, connected-to_serial FROM connections
All items like desktops, laptops, monitors, etc are on the items_list table. To track down the relationship/connections of the items, I created the connections table. IE, Monitor with serial_number=Screen#1 is connected to a Desktop with serial_number=Serial#1. It works ok with my Window Form application because I
used a datagridview control to list all devices simple SQL query.
However, when trying to show the relationship/connection on SQL Reports I've ran out of ideas how to do it. I'm aiming to get the report look like below or something along the lines. I just need to show the connections between the items.
Thank you
You should be able to do this with a table in SSRS if that is what you are using. The query you would need to drive the table of all related items would be:
SELECT item_serial, connected-to_serial, mainItem.*, connectedItem.*
FROM connections
INNER JOIN items_list mainItem ON connections.item_serial = items_list.serial_number
INNER JOIN items_list connectedItem ON connections.connected-to_serial = connectedItem.serial_number
You can of course tailor the SELECT statement to your needs, mainItem.* and connectedItem.* will not give you the most descriptive column names. Using column aliases (found under column_alias here) you can give a more descriptive name to each column.
From here you should be able to use a table and create a row group on the main item (either name or serial number) to get the type of look you are looking to achieve here. I believe the Report Wizard actually has most of the functionality you are looking for and should handle the bulk of this. You may have to move some of the cells around to get the look you are going for though.
Background
I have set up an IoT project using an Azure Event Hub and Azure Stream Analytics (ASA) based on tutorials from here and here. JSON formatted messages are sent from a wifi enabled device to the event hub using webhooks, which are then fed through an ASA query and stored in one of three Azure SQL databases based on the input stream they came from.
The device (Particle Photon) transmits 3 different messages with different payloads, for which there are 3 SQL tables defined for long term storage/analysis. The next step includes real-time alerts, and visualization through Power BI.
Here is a visual representation of the idea:
The ASA Query
SELECT
ParticleId,
TimePublished,
PH,
-- and other fields
INTO TpEnvStateOutputToSQL
FROM TpEnvStateInput
SELECT
ParticleId,
TimePublished,
EventCode,
-- and other fields
INTO TpEventsOutputToSQL
FROM TpEventsInput
SELECT
ParticleId,
TimePublished,
FreshWater,
-- and other fields
INTO TpConsLevelOutputToSQL
FROM TpConsLevelInput
Problem: For every message received, the data is pushed to all three tables in the database, and not only the output specified in the query. The table in which the data belongs gets populated with a new row as expected, while the two other tables get populated with NULLs for columns which no data existed for.
From the ASA Documentation it was my understanding that the INTO keyword would direct the output to the specified sink. But that does not seem to be the case, as the output from all three inputs get pushed to all sinks (all 3 SQL tables).
The test script I wrote for the Particle Photon will send one of each type of message with hardcoded fields, in the order: EnvState, Event, ConsLevels, each 15 seconds apart, repeating.
Here is an example of the output being sent to all tables, showing one column from each table:
Which was generated using this query (in Visual Studio):
SELECT
t1.TimePublished as t1_t2_t3_TimePublished,
t1.ParticleId as t1_t2_t3_ParticleID,
t1.PH as t1_PH,
t2.EventCode as t2_EventCode,
t3.FreshWater as t3_FreshWater
FROM dbo.EnvironmentState as t1, dbo.Event as t2, dbo.ConsumableLevel as t3
WHERE t1.TimePublished = t2.TimePublished AND t2.TimePublished = t3.TimePublished
For an input event of type TpEnvStateInput where the key 'PH' would exist (and not keys 'EventCode' or 'FreshWater', which belong to TpEventInput and TpConsLevelInput, respectively), an entry into only the EnvironmentState table is desired.
Question:
Is there a bug somewhere in the ASA query, or a misunderstanding on my part on how ASA should be used/setup?
I was hoping I would not have to define three separate Stream Analytics containers, as they tend to be rather pricey. After running through this tutorial, and leaving 4 ASA containers running for one day, I used up nearly $5 in Azure credits. At a projected $150/mo cost, there's just no way I could justify sticking with Azure.
ASA is supposed to be purposed for Complex Event Processing. You are using ASA in your queries to essentially pass data from the event hub to tables. It will be much cheaper if you actually host a simple "worker web app" to process the incoming events.
This blog post covers the best practices:
http://blogs.msdn.com/b/servicebus/archive/2015/01/16/event-processor-host-best-practices-part-1.aspx
ASA is great if you are doing some transformations, filters, light analytics on your input data in real-time. Furthermore, it also works great if you have some Azure Machine Learning models that are exposed as functions (currently in preview).
In your example, all three "select into" statements are reading from the same input source, and don't have any filter clauses, so all rows would be selected.
If you only want to rows select specific rows for each of the output, you have to specify a filter condition. For example, assuming you only want records with a non null value in column "PH" for the output "TpEnvStateOutputToSQL", then ASA query would look like below
SELECT
ParticleId,
TimePublished,
PH
-- and other fields INTO TpEnvStateOutputToSQL FROM TpEnvStateInput WHERE PH IS NOT NULL