BigQueryGetDataOperator project ID - google-bigquery

I read the document about BigQueryGetDataOperator and used it. The problem is I didn't know have to pass my select projectID to this operator. So this returns non exist table.
It default use: project_a.dataset_b.table_c.
With I would like to change it to: project_d.dataset_b.table_c.

If you would like to query the table in "project_d", you have to create a new airflow connexion and use a service account created in "project_d" with the proper roles instead of the current connexion using a service account in "project_a".

Related

Update after a Copy Data Activity in Azure Data Factory

I've got this doubt in Azure Data Factory. My pipeline has a copy data activity, and after loading the information in the table I need to update a field in that destination based on a parameter. It is a simple update, but given that we do not have a SQL task (present in SSIS) I do not what to use. Create a SP for this does not seem to be the most appropriate solution, besides, modify the database is complicated. I thought the option "Use Query" in the Lookup activity could be a solution, but this does not allow me to create a SQL query with a parameter, just like in a Source.
What could be a possible workaround?
You are on the right track with the Lookup. That is definitely the way to go. The query field there will allow you to create dynamic SQL just like you did within the copy activity. You just need to reference the variable/parameter properly.
Also, with the Lookup, it will always expect something returned. You don't have to do anything with that returned value. Just ignore it, but the Lookup will not work without returning something. So, that query field would contain something like:
UPDATE dbo.MyTable SET IsComplete = 1 WHERE RunId = #{pipeline().parameters.runId};
SELECT 0 AS DummyValue; -- Necessary for Lookup to work

Thingworx sensor value to Sql Database

I am using Thingworx Platform for IoT. I have connected Thingworx and SQL. I have created 2 database SQL services of the type query and command. Also I have created two tables named Temperature and Humidity.
I am getting Temperature and Humidity values in Thingworx platform. But I am unable to send it to the database, can anyone help? How can I call the properties in the command service?
Database.Conf sql Command code
insert into INFO(Temperature)
values ([[]]);
Thing-Test Subscription Code
var params={Temp:me.Temp_Prop,Hum:me.Hum_Prop};
var result=Things["DatabaseConf"].InsertRecords(params);
When you create a service in ThingWorx you can select the type: "query" is a valid option when you have to insert or retrieve values from a database. You can test the query on SQl and copy/paste into the service.
Check if the thing that connects you to the DataBase has the property "isConnected" equal to True.
Maybe you should also think about how to trigger the Insert service, you can use ValueChange triggers or a Periodic Trigger.
You need an additionnal service of type javascript where you can retrieve the properties, using me.property and invoke the sql services. You can add input parameters to the sql services and use them like this: [[inputParameter]]. In your example that should look like:
insert into INFO(Temperature)
values ([[temp]],[[Hum]]);
If you use the arrow on the right of your input parameter, ThingWorx will write it for you in the proper way already.

SSIS variable filter on different table in non linked Database

I am uploading a data set from excel into out local server that we use to run offline queries and wish to use the information in that table to filter on a table that is in our production system so i can then pull through the filtered table onto out local DB rather than pull the entire table from prod to our local. I was looking at creating a variable that contained all the account numbers in a string and then use that in an IN(?) statement. This worked, but it only picked up the first account in the string and not the remainder.
I am unable to create a temp table in prod or anything as it is locked down for DBA only. I have tried some stuff in SSIS like variable creation etc but it hasn't worked. I am trying to avoid dumping the entire table. If I have to do this I will set it up to update from the valid from field to capture new records.
SELECT
*
FROM account_table
where [account number] in (?)
I have created the ? variable in an separate Execute task SQL box where the ? is
select
string_agg([account number], ', ')
from [distinct account_table].
I have also tried:
select
string_agg([account number], ',')
from [distinct account_table]
but that didn't work.
You can use expressions instead of passing parameters.
Open the Execute SQL Task editor, go to Expressions tab, add a new expression for the SqlStatementSource property as the following:
"SELECT
*
FROM account_table
where [account number] in ('" +
#[User::Variable] + "')"
On the other hand, if i am facing a similar situation i will use lookup transformation to do that instead of SQL statements.
First of all, linked servers is not needed here, it's merely a bridge that connects two servers.
I might be missing something obvious here.. You say that you've loaded the dataset to your local db and now you want to get the data from prod that contains those accounts.
My suggestion is to create a dataflow and use a oledb source to get the data from prod. Then add a lookup that joins to your data set in your local db (join on account, make sure that you connect to your local db and that you redirect no matching rows to no match output ).
Then add a oledb destination to load the data to your local db.

Item in where query source

Hi I'm a newbie to the oracle apex and I'm trying to create a dynamic page that let the user (the logged one) see only "his" record I tried in the source query (P101_USERNAME is the username item on the login page)
select * from user u where u.username=v('P101_USERNAME')
And this
Select * from user u where u.username=:P101_USERNAME
But I got the no data found error I think I'm doing the things the wrong way.. So is there a way to create a page that let the user see only the record linked to his username(or primary key) and hide the other?
That username field is only relevant for the life of the login process.
Throughout the application you should refer to :APP_USER
See list of available substitution strings
https://docs.oracle.com/cd/E59726_01/doc.50/e39147/concept_sub.htm#BEIHCJBG
If you’re using APEX 5, you can now get the User from the APEX$SESSION application context. here
You also can use function APEX_APPLICATION.g_user here

Connect to Database server-side

I've essentially made a MVC application following the tutorial at http://www.asp.net/mvc/tutorials/getting-started-with-mvc3-part1-cs
I can upload it to a server and have the main page run just fine.. but running a different page that interacts with a database brings up the error
" Invalid object name 'dbo.Lyrics'. "
Now I can connect to the database that I'm trying to use (on the server) remotely using management studio just fine.. its called Lyrics and the table is Default.Lyrics ..
The connection string I'm using is "connectionString="Data Source=74.86.97.85;Initial Catalog=Lyrics;User Id=Default;Password=****;""
So my question is .. why is my application trying to use an object with the name "dbo.Lyrics" when my entire application doesn't have that text in it? How can I solve this?
I know that the dbo prefix means DataBase Owner.. and its like a public table.. but since I'm specifying a User ID shouldn't it look for tables with my ID as the prefix?
dbo at the beginning of an object name is a schema. Schemas partition the objects in your database. dbo is simply the default schema.
So, if you have an object named Lyrics, then it's really dbo.Lyrics.