Lets say I have two different landscapes AWS and Google Cloud (Only an example, could be any infrastructure).
Both of them have many variety resources, variables etc. Probably the usage and connection will be also different.
I would like to use both of them in same Terraform script, is it even possible? or every landscape needs it's own script?
Can I maybe add some code (if yes which language is supported?) to identify the landscape and run the relevant resources/providers etc.?
Thanks in advance!
You can define multiple providers in your provider.tf file. I have not tried it across cloud providers but here is the syntax to try.
variable "aws_profile" {}
variable "region" {}
provider "aws" {
alias = "west"
profile = "${var.aws_profile}"
region = "${var.region}"
shared_credentials_file = "~/.aws/credentials"
}
provider "google" {
alias = "central"
credentials = "${file("account.json")}"
project = "my-gce-project"
region = "us-central1"
}
Then you can specify the provider for that resource.
# West coast region
provider "aws" {
alias = "west"
region = "us-west-2"
}
If a provider isn't specified, then the default provider configuration is used (the provider configuration with no alias set). The value of the provider field is TYPE.ALIAS, such as "aws.west" above.
https://www.terraform.io/docs/configuration/providers.html
Edit: terraform load order does not matter.
https://www.terraform.io/docs/configuration/load.html
Related
Consider a wizard generated ASP.NET Core project (NET 6). Add a Google.Cloud.Diagnostics.AspNetCore3 NuGet package and services.AddGoogleDiagnosticsForAspNetCore() to Startup.cs. Let GOOGLE_APPLICATION_CREDENTIALS environment variable point to a path to your service account JSON.
Somewhere in the app (e.g. a controller) add the following:
_logger.LogDebug("Nope");
_logger.LogInformation("Yeah");
Google Cloud Logs Explorer shows only the "Yeah" (no specific filters). My appsettings.json looks like:
"Logging": {
"LogLevel": {
"Default": "Debug",
"System": "Information",
"Microsoft": "Information"
}
}
As far as I understand the "Default": "Debug" should work everywhere where a more specific config is missing.
Why am I not seeing the "Nope" being logged? Anything obvious that I'm missing? It's worth mentioning that both Visual Studio Debug Output as well as the Console output show both Nope/Yeah as expected.
Short Answer: Google.Cloud.Diagnostics.AspNetCore3 does not use appsettings.json (at least for now) and one must explicitly set log levels.
Now to the long answer and working code after that.
To add Google Diagnostics to our project we have 3 overloads of ...AddGoogleDiagnosticsForAspNetCore(...) available, and also ...AddGoogle(...) just to use a service we need, such as logging service. (... at the beginning changes depending on dotnet version, examples at the end).
1- In a GCP environment, ...AddGoogleDiagnosticsForAspNetCore() signature is used to set defaults for the Diagnostics. Service details are fetched from GCP.
2- In a GCP environment, ...AddGoogleDiagnosticsForAspNetCore( AspNetCoreTraceOptions, LoggingServiceOptions, ErrorReportingServiceOptions ) signature we can set 3 types of options: AspNet Tracing, Logging Service and Error Reporting Service.
For this use case, if we want only logging services, we can either use positional arguments (null,new LoggingServiceOptions{...},null) (last null is not required) or named arguments (loggingOptions: new LoggingServiceOptions{...})
There are many to be set in LoggingServicesOptions{...} but just for log level purpose the following will suffice: new LoggingServiceOptions{ Options = LoggingOptions.Create(logLevel: LogLevel.Debug) }.
Now we have come to the important one. Although documentation covers enough of it implicitly, it is not made explicitly clear that this use case will directly set options, not services.
3- Although not explicitly clear, this use is for cases outside GCP or when GCP cannot be set properly(not sure how!?) AddGoogleDiagnosticsForAspNetCore( projectId, serviceName, serviceVersion, TraceOptions, LoggingOptions, ErrorReportingOptions ). This may seem similar to the 2nd signature at first, but it does not set options for services.
When one sees Project ID was not provided and could not be autodetected message for 1st or 2nd signature, they have to provide it as a parameter which immediately switches the function to use this 3rd signature.
In this case, if we want only logging services, it has to be used in the form of (projectId,null,null,null,LoggingOptions...,null) for positional arguments (last null is not required) or (projectId:"some ID",loggingOptions: LoggingOptions...) for named arguments
LoggingOptions... is simply be LoggingOptions.Create(logLevel: LogLevel.Debug) to set log level.
4- Apart from adding these details while adding Google Diagnostics to the services, we can instead add logging options when we set configurations: ...AddGoogle( LoggingServiceOptions{...} ). But in this use, we need to provide a project Id in it; new LoggingServiceOptions{ ProjectId = "some ID", Options = LoggingOptions.Create(logLevel: LogLevel.Debug) }
fill in the ...
dotnet 6 started using new top level statements. so we have following steps to follow.
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddGoogleDiagnosticsForAspNetCore(
projectId: "some ID",
loggingOptions: LoggingOptions.Create(logLevel: LogLevel.Debug)
);
// or
var builder = WebApplication.CreateBuilder(args);
builder.Logging.AddGoogle(
new LoggingServiceOptions {
ProjectId = "some ID",
Options=LoggingOptions.Create(logLevel:LogLevel.Debug)
}
);
Since the OP mentions the use of Startup.cs, the project uses the old style so these are the required parts for that.
// inside ConfigureServices
services.AddGoogleDiagnosticsForAspNetCore(
projectId: "some ID",
loggingOptions: LoggingOptions.Create(logLevel: LogLevel.Debug)
);
// or
// before using "UseStartup"
.ConfigureLogging(
builder => builder.AddGoogle(
new LoggingServiceOptions {
ProjectId = "some ID",
Options=LoggingOptions.Create(logLevel:LogLevel.Debug)
}
)
)
Extra
We can read from the configuration file (top-level format)
var builder = WebApplication.CreateBuilder(args);
var config = builder.Configuration;
builder.Services.AddGoogleDiagnosticsForAspNetCore(
projectId:config["GCP:ID"],
loggingOptions: LoggingOptions.Create(
logLevel: Enum.Parse<LogLevel>(config["GCP:Logging:LogLevel:Default"]
)));
and add a GCP section in appsettings.json
"GCP":{
"ID":"some ID",
"Logging":{
"LogLevel":{
"Default":"Debug"
}
}
}
I've downloaded the mentioned package (it's open-source) and checked default logging-options creation:
As you may see the default logLevel is Information.
And as you go through the implementation there's no sign of reading the level from the config - it's simply passed from the options you may specify in the code:
Initial invocation:
Service provider registration:
Creation of logging provider and options:
Creation of options (1st picture)
And creation of logger (probably invoked somewhere internally by ASP.NET)
The simple answer is Google package doesn't read anything from the appsettings.json by default.
You can set the logging level by using the LoggingOptions:
(Other options omitted for brevity)
builder.Services.AddGoogleDiagnosticsForAspNetCore(loggingOptions: new Google.Cloud.Diagnostics.Common.LoggingServiceOptions
{
// ... Other required options, e.g. projectId
Options = Google.Cloud.Diagnostics.Common
.LoggingOptions.Create(logLevel: LogLevel.Debug
// ... Other necessary options
),
});
I am a complete beginner in using GCP functions/products.
I have written the following code below, that takes a list of cities from a local folder, and call in weather data for each city in that list, eventually uploading those weather values into a table in BigQuery. I don't need to change the code anymore, as it creates new tables when a new week begins, now I would want to "deploy" (I am not even sure if this is called deploying a code) in the cloud for it to automatically run there. I tried using App Engine and Cloud Functions but faced issues in both places.
import requests, json, sqlite3, os, csv, datetime, re
from google.cloud import bigquery
#from google.cloud import storage
list_city = []
with open("list_of_cities.txt", "r") as pointer:
for line in pointer:
list_city.append(line.strip())
API_key = "PLACEHOLDER"
Base_URL = "http://api.weatherapi.com/v1/history.json?key="
yday = datetime.date.today() - datetime.timedelta(days = 1)
Date = yday.strftime("%Y-%m-%d")
table_id = f"sonic-cat-315013.weather_data.Historical_Weather_{yday.isocalendar()[0]}_{yday.isocalendar()[1]}"
credentials_path = r"PATH_TO_JSON_FILE"
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = credentials_path
client = bigquery.Client()
try:
schema = [
bigquery.SchemaField("city", "STRING", mode="REQUIRED"),
bigquery.SchemaField("Date", "Date", mode="REQUIRED"),
bigquery.SchemaField("Hour", "INTEGER", mode="REQUIRED"),
bigquery.SchemaField("Temperature", "FLOAT", mode="REQUIRED"),
bigquery.SchemaField("Humidity", "FLOAT", mode="REQUIRED"),
bigquery.SchemaField("Condition", "STRING", mode="REQUIRED"),
bigquery.SchemaField("Chance_of_rain", "FLOAT", mode="REQUIRED"),
bigquery.SchemaField("Precipitation_mm", "FLOAT", mode="REQUIRED"),
bigquery.SchemaField("Cloud_coverage", "INTEGER", mode="REQUIRED"),
bigquery.SchemaField("Visibility_km", "FLOAT", mode="REQUIRED")
]
table = bigquery.Table(table_id, schema=schema)
table.time_partitioning = bigquery.TimePartitioning(
type_=bigquery.TimePartitioningType.DAY,
field="Date", # name of column to use for partitioning
)
table = client.create_table(table) # Make an API request.
print(
"Created table {}.{}.{}".format(table.project, table.dataset_id, table.table_id)
)
except:
print("Table {}_{} already exists".format(yday.isocalendar()[0], yday.isocalendar()[1]))
def get_weather():
try:
x["location"]
except:
print(f"API could not call city {city_name}")
global day, time, dailytemp, dailyhum, dailycond, chance_rain, Precipitation, Cloud_coverage, Visibility_km
day = []
time = []
dailytemp = []
dailyhum = []
dailycond = []
chance_rain = []
Precipitation = []
Cloud_coverage = []
Visibility_km = []
for i in range(24):
dayval = re.search("^\S*\s" ,x["forecast"]["forecastday"][0]["hour"][i]["time"])
timeval = re.search("\s(.*)" ,x["forecast"]["forecastday"][0]["hour"][i]["time"])
day.append(dayval.group()[:-1])
time.append(timeval.group()[1:])
dailytemp.append(x["forecast"]["forecastday"][0]["hour"][i]["temp_c"])
dailyhum.append(x["forecast"]["forecastday"][0]["hour"][i]["humidity"])
dailycond.append(x["forecast"]["forecastday"][0]["hour"][i]["condition"]["text"])
chance_rain.append(x["forecast"]["forecastday"][0]["hour"][i]["chance_of_rain"])
Precipitation.append(x["forecast"]["forecastday"][0]["hour"][i]["precip_mm"])
Cloud_coverage.append(x["forecast"]["forecastday"][0]["hour"][i]["cloud"])
Visibility_km.append(x["forecast"]["forecastday"][0]["hour"][i]["vis_km"])
for i in range(len(time)):
time[i] = int(time[i][:2])
def main():
i = 0
while i < len(list_city):
try:
global city_name
city_name = list_city[i]
complete_URL = Base_URL + API_key + "&q=" + city_name + "&dt=" + Date
response = requests.get(complete_URL, timeout = 10)
global x
x = response.json()
get_weather()
table = client.get_table(table_id)
varlist = []
for j in range(24):
variables = city_name, day[j], time[j], dailytemp[j], dailyhum[j], dailycond[j], chance_rain[j], Precipitation[j], Cloud_coverage[j], Visibility_km[j]
varlist.append(variables)
client.insert_rows(table, varlist)
print(f"City {city_name}, ({i+1} out of {len(list_city)}) successfully inserted")
i += 1
except Exception as e:
print(e)
continue
In the code, there is direct reference to two files that is located locally, one is the list of cities and the other is the JSON file containing the credentials to access my project in GCP. I believed that uploading these files in Cloud Storage and referencing them there won't be an issue, but then I realised that I can't actually access my Buckets in Cloud Storage without using the credential files.
This leads me to being unsure whether the entire process would be possible at all, how do I authenticate in the first place from the cloud, if I need to reference that first locally? Seems like an endless circle, where I'd authenticate from the file in Cloud Storage, but I'd need authentication first to access that file.
I'd really appreciate some help here, I have no idea where to go from this, and I also don't have great knowledge in SE/CS, I only know Python R and SQL.
For Cloud Functions, the deployed function will run with the project service account credentials by default, without needing a separate credentials file. Just make sure this service account is granted access to whatever resources it will be trying to access.
You can read more info about this approach here (along with options for using a different service account if you desire): https://cloud.google.com/functions/docs/securing/function-identity
This approach is very easy, and keeps you from having to deal with a credentials file at all on the server. Note that you should remove the os.environ line, as it's unneeded. The BigQuery client will use the default credentials as noted above.
If you want the code to run the same whether on your local machine or deployed to the cloud, simply set a "GOOGLE_APPLICATION_CREDENTIALS" environment variable permanently in the OS on your machine. This is similar to what you're doing in the code you posted; however, you're temporarily setting it every time using os.environ rather than permanently setting the environment variable on your machine. The os.environ call only sets that environment variable for that one process execution.
If for some reason you don't want to use the default service account approach outlined above, you can instead directly reference it when you instantiate the bigquery.Client()
https://cloud.google.com/bigquery/docs/authentication/service-account-file
You just need to package the credential file with your code (i.e. in the same folder as your main.py file), and deploy it alongside so it's in the execution environment. In that case, it is referenceable/loadable from your script without needing any special permissions or credentials. Just provide the relative path to the file (i.e. assuming you have it in the same directory as your python script, just reference only the filename)
There may be different flavors and options to deploy your application and these will depend on your application semantics and execution constraints.
It will be too hard to cover all of them and the official Google Cloud Platform documentation cover all of them in great details:
Google Compute Engine
Google Kubernetes Engine
Google App Engine
Google Cloud Functions
Google Cloud Run
Based on my understanding of your application design, the most suitable ones would be:
Google App Engine
Google Cloud Functions
Google Cloud Run: Check these criteria to see if you application is a good fit for this deployment style
I would suggest using Cloud Functions as you deployment option in which case your application will default to using the project App Engine service account to authenticate itself and perform allowed actions. Hence, you should only check if the default account PROJECT_ID#appspot.gserviceaccount.com under the IAM configuration section has proper access to needed APIs (BigQuery in your case).
In such a setup, you want need to push your service account key to Cloud Storage which I would recommend to avoid in either cases, and you want need to pull it either as the runtime will handle authentication the function for you.
I am currently evaluating ImageFlow Server (https://github.com/imazen/imageflow-dotnet-server) to determine if it will meet the needs of a project that I am working on. Working through the documentation, I was able to get the ImageFlow Server connected to Azure Storage using the following:
public void ConfigureServices(IServiceCollection services)
{
services.AddImageflowAzureBlobService(
new AzureBlobServiceOptions("[MY CONNECTION STRING TO AZURE STORAGE]",
new BlobClientOptions())
.MapPrefix("/azure", "[CONTAINER No. 1]"));
}
This works without issue and I can see images as expected. Current requirements for the project requires that each user will have a unique container though, which makes the implementation above impossible.
Is there a way to pass the container name along with the file name when making a request? Something like: '/azure/CONTAINER/image.jpg?w=250'
We have an example provider to do exactly that here: https://github.com/imazen/imageflow-dotnet-server/blob/main/examples/Imageflow.Server.Example/CustomBlobService.cs
// Custom blob services can do whatever you need. See CustomBlobService.cs in src/Imageflow.Service.Example
services.AddImageflowCustomBlobService(new CustomBlobServiceOptions()
{
Prefix = "/custom_blobs/",
IgnorePrefixCase = true,
ConnectionString = "UseDevelopmentStorage=true;",
// Only allow 'my_container' to be accessed. /custom_blobs/my_container/key.jpg would be an example path.
ContainerKeyFilterFunction = (container, key) =>
container == "my_container" ? Tuple.Create(container, key) : null
});
Lets say I have applicationA that has 3 property files:
-> applicationA
- datasource.properties
- security.properties
- jms.properties
How do I move all properties to a spring cloud config server and keep them separate?
As of today I have configured the config server that will only read ONE property file as this seems to be the standard way. This file the config server picks up seems to be resolved by using the spring.application.name. In my case it will only read ONE file with this name:
-> applicationA.properties
How can I add the other files to be resolved by the config server?
Not possible in the way how you requested. Spring Cloud Config Server uses NativeEnvironmentRepository which is:
Simple implementation of {#link EnvironmentRepository} that uses a SpringApplication and configuration files located through the normal protocols. The resulting Environment is composed of property sources located using the application name as the config file stem (spring.config.name) and the environment name as a Spring profile.
See: https://github.com/spring-cloud/spring-cloud-config/blob/master/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/NativeEnvironmentRepository.java
So basically every time when client request properties from Config Server it creates ConfigurableApplicationContext using SpringApplicationBuilder. And it is launched with next configuration property:
String config = application;
if (!config.startsWith("application")) {
config = "application," + config;
}
list.add("--spring.config.name=" + config);
So possible names for property files will be only application.properties(or .yml) and config client application name that is requesting configuration - in your case applicationA.properties.
But you can "cheat".
In config server configuration you can add such property
spring:
cloud:
config:
server:
git:
search-paths: '{application}, {application}/your-subdirectory'
In this case Config Server will search for same property file names but in few directories and you can use subdirectories to keep your properties separate.
So with configuration above you will be able to load configuration from:
applicationA/application.properies
applicationA/your-subdirectory/application.properies
This can be done.
You need to create your own EnvironmentRepository, which loads your property files.
org.springframework.cloud.config.server.support.AbstractScmAccessor#getSearchLocations
searches for the property files to load :
for (String prof : profiles) {
for (String app : apps) {
String value = location;
if (app != null) {
value = value.replace("{application}", app);
}
if (prof != null) {
value = value.replace("{profile}", prof);
}
if (label != null) {
value = value.replace("{label}", label);
}
if (!value.endsWith("/")) {
value = value + "/";
}
output.addAll(matchingDirectories(dir, value));
}
}
There you could add custom code, that reads the required property files.
The above code matches exactly the behaviour described in the spring docs.
The NativeEnvironmentRepository does NOT access GIT/SCM in any way, so you should use
JGitEnvironmentRepository as base for your own implementation.
As #nmyk pointed out, NativeEnvironmentRepository boots a mini app in order to collect the properties by providing it with - sort of speak - "hardcoded" {appname}.* and application.* supported property file names. (#Stefan Isele - prefabware.com JGitEnvironmentRepository ends up using NativeEnvironmentRepository as well, for that matter).
I have issued a pull request for spring-cloud-config-server 1.4.x, that supports defining additional file names, through a spring.cloud.config.server.searchNames environment property, in the same sense one can do for a single springboot app, as defined in the Externalized Configuration.Application Property Files section of the documentation, using the spring.config.name enviroment property. I hope they review it soon, since it seems many have asked about this feature in stack overflow, and surely many many more search for it and read the currently advised solutions.
It worths mentioning that many ppl advise "abusing" the profile feature to achieve this, which is a bad practice, in my humble opinion, as I describe in this answer
I am performing a query using the BigQuery Java API with the following code:
try (FileInputStream input = new FileInputStream(serviceAccountKeyFile)) {
GoogleCredentials credentials = GoogleCredentials.fromStream(input);
BigQuery bigQuery = BigQueryOptions.newBuilder()
.setCredentials(credentials)
.build()
.getService();
QueryRequest request = QueryRequest.of("SELECT * FROM foo.Bar");
QueryResponse response = bigQuery.query(request);
// Handle the response ...
}
Notice that I am using a specific service account whose key file is given by serviceAccountKeyFile.
I was expecting that the API would pick up the project_id from the key file. But it is actually picking up the project_id from the default key file referenced by the GOOGLE_APPLICATION_CREDENTIALS environment variable.
This seems like a bug to me. Is there a way to workaround the bug by setting the default project explicitly?
Yeah, that doesn't sound right at all. It does sound like a bug. I always just use the export the GOOGLE_APPLICATION_CREDENTIALS environment variable in our applications.
Anyway, you try explicitly setting the project id to see if it works:
BigQuery bigQuery = BigQueryOptions.newBuilder()
.setCredentials(credentials)
.setProjectId("project-id") //<--try setting it here
.build()
.getService();
I don't believe the project is coming from GOOGLE_APPLICATION_CREDENTIALS. I suspect that the project being picked up is the gcloud default project set by gcloud init or gcloud config set project.
From my testing, BigQuery doesn't use a project where the service account is created. I think the key is used only for authorization, and you always have to set a target project. There are a number of ways:
.setProjectId(<target-project>) in the builder
Define GOOGLE_CLOUD_PROJECT
gcloud config set project <target-project>
The query job will then be created in target-project. Of course, your service key should have access to target-project, which may or may not be the same project where your key is created. That is, you can run a query on projects other than the project where your key is created, as long as your key has permission to do so.