Situation:
I am deploying a Synapse workspace instance in Bicep with Managed Virtual Network Enabled.
I can see the Managed Vnet Is enabled from the UI:
However, when I enter the workspace my integration runtimes are not enabled for virtual network access and I cannot create managed private endpoints.
I'm writing the following code for the bicep deployment:
resource synapse_workspace 'Microsoft.Synapse/workspaces#2021-06-01' = {
name: synapse_workspace_name
location: location
tags: {
Workload: '####'
Environment: envName
Classification: 'Confidential'
Criticality: 'Low'
}
identity: {
type: 'SystemAssigned'
}
properties: {
// Git Repo
workspaceRepositoryConfiguration: {
accountName: '#####'
collaborationBranch: 'main'
projectName: '####'
repositoryName: '#############'
rootFolder: '/synapse/syn-data-${envName}'
tenantId: '####################'
type: 'WorkspaceVSTSConfiguration'
}
defaultDataLakeStorage: {
resourceId: storage_account_id
createManagedPrivateEndpoint: true
accountUrl: ###################
filesystem: ################
}
encryption: {
cmk: {
kekIdentity: {
useSystemAssignedIdentity: true
}
key: {
name: 'default'
keyVaultUrl: '#########################'
}
}
}
managedVirtualNetwork: 'default'
connectivityEndpoints: {
web: 'https://web.azuresynapse.net?workspace=%2fsubscriptions%######################
dev: 'https://##############.dev.azuresynapse.net'
sqlOnDemand: '################-ondemand.sql.azuresynapse.net'
sql: '################.sql.azuresynapse.net'
}
managedResourceGroupName: guid('synapseworkspace-managed-resource-group-${envName}')
sqlAdministratorLogin: 'sqladminuser'
privateEndpointConnections: []
managedVirtualNetworkSettings: {
preventDataExfiltration: true
allowedAadTenantIdsForLinking: []
}
publicNetworkAccess: 'Disabled'
cspWorkspaceAdminProperties: {
initialWorkspaceAdminObjectId: '#########################'
}
trustedServiceBypassEnabled: false
}
}
I get no errors in the deployment regarding the virtual network or any associated settings, but I still get the default integration runtime set to "Public" and not "Managed Virtual Network".
Is this a limitation in Bicep or am I missing some parameter?
Any help would be great
Joao
Related
I am trying to setup an event Notification system on s3 to publish notifications to SNS when a file is being uploaded to s3. Here 's how I implemented it via CDK :
import * as sns from "monocdk/aws-sns";
import * as iam from "monocdk/aws-iam";
import {
GAMMA_ACCOUNT,
PROD_ACCOUNT,
UAT1_ACCOUNT,
UAT2_ACCOUNT,
PERFECT_MILE_ACCOUNT,
} from "../utils/constants/awsAccounts";
import { Construct } from "monocdk";
import * as s3 from "monocdk/aws-s3";
import * as s3n from "monocdk/aws-s3-notifications";
import { CommonResourceStackProps, Stage } from "../stack/CommonResourcesStack";
export class S3NotificationToSNSCustomResource extends Construct {
constructor(
scope: Construct,
id: string,
bucket: s3.IBucket,
stackProps: CommonResourceStackProps
) {
super(scope, id);
const topic = new sns.Topic(this, "Topic", {
displayName: "Sherlock-s3-Event-Notifications-Topic",
topicName: "Sherlock-s3-Event-Notifications-Topic",
});
const topicPolicy = new sns.TopicPolicy(this, "TopicPolicy", {
topics: [topic],
});
const s3ServicePrincipal = new iam.ServicePrincipal("s3.amazonaws.com");
topicPolicy.document.addStatements(
new iam.PolicyStatement({
sid: "0",
actions: ["sns:Publish"],
principals: [s3ServicePrincipal],
resources: [topic.topicArn],
conditions: {
StringEquals: {
"AWS:SourceOwner":
stackProps.stage == Stage.Prod
? PROD_ACCOUNT
: stackProps.stage == Stage.Gamma
? GAMMA_ACCOUNT
: stackProps.stage == Stage.UAT1
? UAT1_ACCOUNT
: UAT2_ACCOUNT,
},
ArnLike: { "AWS:SourceArn": bucket.bucketArn },
},
}),
new iam.PolicyStatement({
sid: "1",
actions: ["sns:Subscribe"],
principals: [new iam.AccountPrincipal(PERFECT_MILE_ACCOUNT)],
resources: [topic.topicArn],
})
);
bucket.addEventNotification(
s3.EventType.OBJECT_CREATED,
new s3n.SnsDestination(topic),
{ prefix: "output/reportingData/openItems/", suffix: "_SUCCESS" }
);
}
}
But, when I try to deploy this, I am getting the following error : An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations
Can anyone help me with it?
I read this post(https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/) but its resolution is using the templates and I am implementing using the CDK package. Also I have added all the access policies to publish and subscribe.
aws:SourceAccount and aws:SourceOwner are condition keys which are not supported by all services. Amazon S3 notifications use aws:SourceAccount Refer - https://docs.aws.amazon.com/sns/latest/dg/sns-access-policy-use-cases.html#source-account-versus-source-owner
Storage account deployed from ARMTemplate is creating diagnostic settings as disabled.
How to enable diagnostics status using ARMTemplate or Powershell script?
Want to automate the process to deploy diagnostic settings.
Here is a solution using ARM templates in the newer Bicep format. In the example, it configures diagnostics settings for:
StorageAccount
Blob
File
Queue
Table
To reduce the template length, it configures only the StorageRead on the storage account services.
param name string
param location string = resourceGroup().location
param sku string
#description('Resource ID for the destination log analytics workspace.')
param logAnalyticsWorkspaceId string
resource storageAccount 'Microsoft.Storage/storageAccounts#2019-06-01' = {
name: name
location: location
kind: 'StorageV2'
sku: {
name: sku
}
properties: {
allowBlobPublicAccess: false
allowSharedKeyAccess: true
minimumTlsVersion: 'TLS1_2'
accessTier: 'Hot'
supportsHttpsTrafficOnly: true
networkAcls: {
defaultAction: 'Deny'
bypass: 'AzureServices'
}
}
}
resource diagnosticsStorage 'Microsoft.Insights/diagnosticSettings#2021-05-01-preview' = {
scope: storageAccount
name: 'diagnostics00'
properties: {
workspaceId: logAnalyticsWorkspaceId
metrics: [
{
category: 'Transaction'
enabled: true
}
]
}
}
resource blobService 'Microsoft.Storage/storageAccounts/blobServices#2021-06-01' = {
parent: storageAccount
name: 'default'
properties: {}
}
resource diagnosticsBlob 'Microsoft.Insights/diagnosticSettings#2021-05-01-preview' = {
scope: blobService
name: 'diagnostics00'
properties: {
workspaceId: logAnalyticsWorkspaceId
logs: [
{
category: 'StorageRead'
enabled: true
}
]
}
}
resource fileService 'Microsoft.Storage/storageAccounts/fileServices#2021-06-01' = {
parent: storageAccount
name: 'default'
properties: {}
}
resource diagnosticsFile 'Microsoft.Insights/diagnosticSettings#2021-05-01-preview' = {
scope: fileService
name: 'diagnostics00'
properties: {
workspaceId: logAnalyticsWorkspaceId
logs: [
{
category: 'StorageRead'
enabled: true
}
]
}
}
resource queueService 'Microsoft.Storage/storageAccounts/queueServices#2021-06-01' = {
parent: storageAccount
name: 'default'
properties: {}
}
resource diagnosticsQueue 'Microsoft.Insights/diagnosticSettings#2021-05-01-preview' = {
scope: queueService
name: 'diagnostics00'
properties: {
workspaceId: logAnalyticsWorkspaceId
logs: [
{
category: 'StorageRead'
enabled: true
}
]
}
}
resource tableService 'Microsoft.Storage/storageAccounts/tableServices#2021-06-01' = {
parent: storageAccount
name: 'default'
properties: {}
}
resource diagnosticsTable 'Microsoft.Insights/diagnosticSettings#2021-05-01-preview' = {
scope: tableService
name: 'diagnostics00'
properties: {
workspaceId: logAnalyticsWorkspaceId
logs: [
{
category: 'StorageRead'
enabled: true
}
]
}
}
Please follow the below URL to enable the Diagnostics Settings for Azure Storage Account using ARM Template:
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/resource-manager-diagnostic-settings#diagnostic-setting-for-azure-storage
I am trying to use hapi-sequelize 3.0.4 with hapi.js 17.2.0 using the following configuration:
{
plugin: require("hapi-sequelize"),
options:[
{
name: "mysql", // identifier
models: ["./plugins/sequelize/models/*.js"], // paths/globs to model files
sequelize: new Sequelize(
process.env.DB_NAME,
process.env.DB_USER,
process.env.DB_PASS,
{
host: process.env.DB_HOST,
dialect: "mysql",
port: process.env.DB_PORT ? parseInt(process.env.DB_PORT) : 3306
}
), // sequelize instance
sync: false, // sync models - default false
forceSync: false // force sync (drops tables) - default false
}
]
}
It gives the following error:
name[1] --missing---
Is this the latest version of hapi-sequelize compatible with hapi.js 17.2.0?
I created a second manager for my second database.
In this base, i create a table which it contains my users.
The problem is the symfony don't load users from this database.
here is an extract of my config.yml :
Doctrine Configuration
doctrine:
dbal:
default_connection: default
connections:
default:
driver: pdo_mysql
host: "%database_host%"
port: "%database_port%"
dbname: "%database_name%"
user: "%database_user%"
password: "%database_password%"
charset: UTF8
seconddb:
driver: pdo_mysql
host: "xx.xx.xx.xx"
port: "3306"
dbname: "acme_test"
user: "acmegamestest"
password: "mypassword"
charset: UTF8
# if using pdo_sqlite as your database driver:
# 1. add the path in parameters.yml
# e.g. database_path: "%kernel.root_dir%/data/data.db3"
# 2. Uncomment database_path in parameters.yml.dist
# 3. Uncomment next line:
# path: "%database_path%"
orm:
default_entity_manager: default
auto_generate_proxy_classes: "%kernel.debug%"
#naming_strategy: doctrine.orm.naming_strategy.underscore
entity_managers:
default:
connection: default
mappings:
acmeAdminBundle: ~
acmeBlogBundle: ~
gedmo_translatable:
type: annotation
alias: GedmoTranslatable
prefix: Gedmo\Translatable\Entity
is_bundle: false
# make sure vendor library location is correct
dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Translatable/Entity"
seconddb:
connection: seconddb
mappings:
acmeJoueurBundle: ~
and here is an extract of my security.yml :
encoders:
FOS\UserBundle\Model\UserInterface: sha512
Acme\JoueurBundle\Entity\Players:
algorithm: sha512
encode_as_base64: false
providers:
seconddb:
entity:
class: Acme\JoueurBundle\Entity\Players
property: username
manager_name: seconddb
fos_userbundle:
id: fos_user.user_provider.username
firewalls:
administration:
pattern: ^/admin
provider: fos_userbundle
context: administration
form_login:
#csrf_provider: security.csrf.token_manager
login_path : fos_user_security_login
check_path : fos_user_security_check
failure_path : null
default_target_path : /admin
logout:
path : fos_user_security_logout
target : /connexion
anonymous: true
frontend:
pattern: ^/
provider: acme_joueurbundle
context: frontend
form_login:
#csrf_provider: form.csrf_provider
login_path : acme_players_login
check_path : acme_players_check
failure_path : null
default_target_path : acme_players_userprofile
logout:
path : acme_players_logout
target : acme_players_login
anonymous: true
and my entity implements "AdvancedUserInterface, \Serializable"
with this function :
//////////////////////////liaison pour symfony////////////////////////////
public function getRoles()
{
return array('ROLE_PLAYERS');
}
public function getSalt(){
return $this->salt;
}
public function eraseCredentials(){
}
public function isAccountNonExpired()
{
return true;
}
public function isCredentialsNonExpired()
{
return true;
}
public function isAccountNonLocked()
{
return !$this->banned;
}
public function isEnabled()
{
return $this->active;
}
/** #see \Serializable::serialize() */
public function serialize()
{
return serialize(array(
$this->id,
$this->username,
$this->password,
$this->active
// see section on salt below
// $this->salt,
));
}
/** #see \Serializable::unserialize() */
public function unserialize($serialized)
{
list (
$this->id,
$this->username,
$this->password,
$this->active
// see section on salt below
// $this->salt
) = unserialize($serialized);
}
/////////////////////////////////////////////////////////////////////////////
However when i trying to connect, i get this errors in dump function :
and in the section Doctrine i get this :
Is what you know the solution?
Hope you can help me.
Thanks
EDIT
So, the error comes from this piece of code in the user authentication provider :
try {
$user = $this->retrieveUser($username, $token);
} catch (UsernameNotFoundException $e) {
if ($this->hideUserNotFoundExceptions) {
throw new BadCredentialsException('Bad credentials.', 0, $e);
}
$e->setUsername($username);
throw $e;
}
The method retrieveUser failed and it throws bad crendentials exception. This explains why there is no connection since it has not yet been called.
Before asking you for the code of your provider, let's look at your security.yml because you create a provider called seconddb and in your firewall frontend you try to call a provider acme_joueurbundle who propably doesn't exist.
So, your security.yml should look like this:
# app/config/security.yml
[...]
providers:
seconddb:
entity:
class: Acme\JoueurBundle\Entity\Players
property: username
manager_name: seconddb
fos_userbundle:
id: fos_user.user_provider.username
firewalls:
administration:
provider: fos_userbundle
[...]
frontend:
provider: seconddb
[...]
I am trying to connect logstash with elasticsearch through authentication but this configuration gives me the following error : [401] Forbidden by ReadonlyREST ES plugin {:class=>"Elasticsearch::Transport::Transport::Errors::Unauthorized", :level=>:error}
Configuration files are given below:
[Elasticsearch conf file]
http.cors.enabled: true
http.cors.allow-origin: /https?:\/\/localhost(:[0-9]+)?/
readonlyrest:
enable: true
response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin
access_control_rules:
- name: "Logstash can write and create its own indices"
auth_key: logstash:logstash
type: allow
actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
indices: ["logstash-*", "<no_index>"]
- name: Kibana Server (we trust this server side component, full access granted via HTTP authentication)
auth_key: admin:pass3
type: allow
- name: Developer (reads only logstash indices, but can create new charts/dashboards)
auth_key: dev:dev
type: allow
kibana_access: ro+
indices: ["<no-index>", ".kibana*", "logstash*", "default"]
[logstash conf file]
input {
file {
path =>"/var/log/site.log"
start_position => beginning
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
user => "logstash"
password => "logstash"
}
}
Mention output in logstash config file like below :-
output {
elasticsearch {
hosts => ["localhost:9200"]
user => ["logstash"]
password => ["logstash"]
}