I have 5 BigQuery projects namely Project1, Project2… Project5.
I have a service account created in Project1. I am using this same service account in all the 5 projects for programatic insertion of rows.
Now I plan to delete Project1 for some reasons. My question is whether the service account that exits in Project1 also get deleted?
Since the other 4 projects use this service account, will the insertion to these projects fail?
If the service account gets deleted, is there a way to migrate this service account to any of the other project.
I don't want to create a new pair since the projects are in production and requires roll outs to handle the newly generated keys. Any suggestions in this regard.
Service accounts gets deleted when the project that contains it is deleted, so i am not sharing it across projects.
So I created separate service accounts per big-query project for my use case.
Related
I need to query a BigQuery table in one GCP project (say #1) and one location (EU) and write results to a table in another project (say #2) and another location (US) with Airflow.
Composer/Airflow instance itself runs in project #2 and location US.
Airflow is using GCP connection configured with a service account from project #2 which also has most of the rights in project #1.
I realise that this might involve multiple extra steps such as storing data temporarily in GCS, so this is fine as long as the end result is achieved.
How should I approach this problem? I saw quite a few articles but none does suggest a strategy for dealing with this situation which I suppose is fairly common.
Say that I have two projects in GCP, Projects A and B. Project A has a BigQuery table, and project B needs to read that data using the BigQuery Storage Read API.
If I create a service account in project B, then go to project A and assign this user the role BigQuery Read Session User as well as add them to the dataset ACL, they will be able to stream the table content. Which project will receive the bill for the data extracted? Project A where the read session is created, or Project B which is the home of the acting service account?
To be clear, I would like for Project B to pay for the load they generate.
I have tried to find a way to be explicit about this, but as far as I can tell there is no way to specify billing project when creating a read session. I have also checked what happens when I try to create a read session with the "parent project" set to Project B while the table location says Project A, and this just leads to the table not being found at all.
In Storage Read API pricing, BigQuery charges for the number of bytes processed (also referred to as bytes read). In your scenario, Project A has the Bigquery table and where the Read Session happens and you just attached Project B's service account as BigQuery Read Session User in Project A, hence the billed amount will go to Project A.
As an alternative, you can check using Billing Reports the cost trends for Bigquery Storage API with the following filters:
Newly upgraded TFS server has two user accounts created, "Project Collection Build Service" and "Project Collection Build Service (Team Foundation)". Both accounts have the same GUID (listed in the Username or Scope column). These were created by TFS and the GUID begins with "Build\" Attempts to delete either results in an error. Using TFSConfig to list user accounts, these do not show up.
To run builds, the Project Collection Build Service Accounts group must have, as a member, the Project Collection Build Service user account. Both of the accounts listed above can't be added to the group because of the matching GUIDs.
Every few hours, builds start failing with error stating
The workspace xxxx;Project Collection Build Service does not exist.
This means that the user account with the added (TEAM FOUNDATION) is in the group. If it is removed and the other one is added, builds start working again. Until a few hours later when the builds begin to fail with error stating
The workspace xxxx;Project Collection Build Service (TEAM FOUNDATION) does not exist.
Swap them again and builds start working. It appears it is applying the user account via the GUID but then doing some kind of verification based on the Display Name.
Some help pinpointing this error came from the post at http://www.codewrecks.com/blog/index.php/2016/01/15/troubleshoot-error-tf140
44-in-build-vnext-for-tfs2015/ but I can't find any other references to an issue with these accounts.
You may try to use tfssecurity /gd command to delete a server-level or collection-level group:
tfssecurity /gd groupIdentity [/collection:CollectionURL] [/server:ServerURL]
The group identity is the security identifier (SID). For more information about finding the SID of a group, see /im: Display information about identities that compose direct membership. You can also use the friendly name to delete a group.
Finally found solution at this link:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/495e59d2-d3e3-432d-be
98-1f0c358c2bc2/tf14061-the-workspace-xxxx-does-not-exist?forum=tfsbuild
After deleting all Agents and associated workspaces then adding a new agent and rebooting the server we have had no problems.
I know if any user query data from a dataset, that person will get billed for the query, while the project will get billed for the storage. Is it possible to set it up so the project or billing account which creates the project get billed for the query instead of the person who did the query?
I guess one solution to this is to create a service account and have that service account does the querying through a web app.
Who is billed for BigQuery queries?
it is not the user per se who is being billed for querying data - it is rather project (from which query is being executed) get's billed. Of course if that given user happen to be a billing owner - it means that the user gets billed.
The only way I know to change this is:
be added (as Project Viewer at least ) to the project that queryable data is in
log into that project and execute query from within that project
Note if you are in Web UI - the active project becomes a billing project. and if you are using bq command line - you need to set default or billing project using respectve flag
My questions is as follows:
Is there a way to make a master project plan incorporating multiple projects (not necessarily sub projects) in Microsoft office. As our resources get pushed around due to parts and items not arriving and shipping dates being moved forward amongst other things etc, sometimes the resource plan has to be changed regularly. I want to be able to pull all current projects into a master project plan so that I can identify where project resources are overlapping. Not necessarily by task but more by employee.
To try and explain a bit better:
Project 1:
Project Manager: John
Project Engineer: Jack. Task - Drawing
John assigns Jack to work on a task in Project 1.
Project 2:
Project Manager: Mark
Project Engineer: Jack. Task - Documentation
Jack wasn't supposed to be working on Project 2 for a further 2 weeks but the deadline has been moved forward and Mark has also assigned Jack to work on a task on his project.
I'd like to create a master project where I can pull in Project 1 and Project 2 and find a way for it to identify the resource overlap, regardless of the fact that Jack has been assigned to 2 different tasks, but more because he as an employee has been assigned to two projects at once.
Is this possible?
On a larger scale realistically I'll need this to incorporate about 6 Projects and about 20 staff members across, so I can find all the overlaps.
I am aware that there is a way to split a person between 2 tasks by assigning a percentage that they will work on both, i.e 90% on task 1 and 10% on task 2 but obviously this won't be project exclusive and my aim is to identify the overlaps rather than create them on purpose for resource sharing.
You are basically looking for software to support your "resource leveling" process.
The ]project-open[ open-source PPM software is capable of importing MS-Project schedules. After the import, you can get Resource Management reports from the system that allow you to perform manual resource leveling.
Another interesting open-source tool is TaskJuggler. TJ actually does multi-project scheduling or automatic resource leveling. However, TJ does not include a MS-Project integration at the moment AFAIK.
Affiliation note: I'm a member of the ]project-open[ team.
Depending on how you want to approach this, you can have a single master project and make all of your other project files sub projects in that master file. You can work in this master file and it will save your updates to the other files. It is sort of like having Project server on your desktop. You can create veiws for each project, or simply roll them up and they show as a single summary task.
You can do this by using a shared resource file but the best way now would be to just get a Project Online subscription and do your projects there.