I have a bucket (logs) in Amazon S3 (us-east-1) with, unsurprisingly, logs, partitioned by application and date:
logs
├── peacekeepers
│ └── year=2018
│ ├── month=11
│ │ ├── day=01
│ │ ├── day=…
│ │ └── day=30
│ └── month=12
│ ├── day=01
│ ├── day=…
│ └── day=19
│ ├── 00:00 — 01:00.log
│ ├── …
│ └── 23:00 — 00:00.log
├── rep-hunters
├── retro-fans
└── rubber-duckies
I want to list all the objects (logs) for a particular date, month, year…
How do I do that with AWS SDK for Java 2.x?
New SDK makes it easy to work with paginated results:
S3Client client = S3Client.builder().region(Region.US_EAST_1).build();
ListObjectsV2Request request =
ListObjectsV2Request
.builder()
.bucket("logs")
.prefix("peacekeepers/year=2018/month=12")
// .prefix("peacekeepers/year=2018/month=12/day=19")
.build();
ListObjectsV2Iterable response = client.listObjectsV2Paginator(request);
for (ListObjectsV2Response page : response) {
for (S3Object object : page.contents()) {
// Consume the object
System.out.println(object.key());
}
}
Related
I have somewhat below folder structure:
.
├── locals.tf
├── main.tf
├── modules
│ ├── bigquery
│ │ ├── main.tf
│ │ ├── schema
│ │ └── variables.tf
│ ├── bigquery_tables
│ │ ├── main.tf
│ │ ├── schema
│ │ └── variables.tf
│ ├── bigquery_views
│ │ ├── main.tf
│ │ ├── queries
│ │ ├── schema
│ │ └── variables.tf
│ ├── cloud_composer
│ ├── list_projects
├── providers.tf
├── storage.tf
├── variables.tf
└── versions.tf
& my main.tf in bigquery_tables is:
resource "google_bigquery_table" "bq_tables" {
for_each = { for table in var.bigquery_dataset_tables : table.table_id => table }
project = var.project_id
dataset_id = each.value.dataset_id
table_id = each.value.table_id
schema = file(format("${path.module}/schema/%v.json", each.value.file_name))
deletion_protection = false
dynamic "time_partitioning" {
for_each = try(each.value.time_partition, false) == false ? [] : [each.value.time_partition]
content {
type = each.value.partitioning_type
field = each.value.partitioning_field
}
}
}
The issue I am facing is since things are in development stage we have frequent update in schema.
Thus this change in schema is causing the BigQuery table to be replaced by terraform & also leading to data loss in BigQuery.
Can some one suggest a solution like what I should add in by resource block to avoid replacing the table & data loss?
I am unsure how can I add "external_data_configuration" in my current block as per https://github.com/hashicorp/terraform-provider-google/issues/10919.
Unfortunately if you change the schema structure of the BigQuery table, you will have this behaviour.
For the deletion_protection param, I think it's better to set it to true to prevent data loss or not set it (it' true by default).
The solution I saw, because you are in dev mode, it's using the Terraform workspace and run your schema updates in a separated workspace.
It will create a new dataset or table in each apply, example :
locals.tf file :
locals {
workspace = terraform.workspace != "default" ? "${terraform.workspace}_" : ""
}
main.tf file :
resource "google_bigquery_table" "bq_tables" {
for_each = { for table in var.bigquery_dataset_tables : table.table_id => table }
project = var.project_id
dataset_id = "${local.workspace}${each.value.dataset_id}"
table_id = each.value.table_id
schema = file(format("${path.module}/schema/%v.json", each.value.file_name))
deletion_protection = false
dynamic "time_partitioning" {
for_each = try(each.value.time_partition, false) == false ? [] : [each.value.time_partition]
content {
type = each.value.partitioning_type
field = each.value.partitioning_field
}
}
}
In this example, I used the workspace as prefix of dataset ID, if the default workspace is used, the prefix is empty, otherwise equals to the given workspace :
dataset=mydataset
- workspace=default => dataset=mydataset
- workspace=evolfeature1 => dataset=evolfeature1_mydataset
Let's say I have two packages in a monorepo using pnpm. One for Vue#3 (package-a), one other for Vue#2 (package-b). For the record, they both depend on a third package called core.
You can find what it looks like below:
.
├── .npmrc
├── .nvmrc
├── package.json
├── packages
│ ├── package-a
│ │ ├── index.ts
│ │ ├── node_modules
│ │ │ ├── .bin
│ │ │ │ ├── tsc
│ │ │ │ ├── tsserver
│ │ │ │ ├── vite
│ │ │ │ └── vue-tsc
│ │ │ ├── #myscope
│ │ │ │ └── core -> ../../../core
│ │ │ ├── #vitejs
│ │ │ │ └── plugin-vue -> ../../../../node_modules/.pnpm/#vitejs+plugin-vue#2.3.3_vite#2.9.9+vue#3.2.36/node_modules/#vitejs/plugin-vue
│ │ │ ├── typescript -> ../../../node_modules/.pnpm/typescript#4.6.4/node_modules/typescript
│ │ │ ├── vite -> ../../../node_modules/.pnpm/vite#2.9.9/node_modules/vite
│ │ │ ├── vue -> ../../../node_modules/.pnpm/vue#3.2.36/node_modules/vue
│ │ │ └── vue-tsc -> ../../../node_modules/.pnpm/vue-tsc#0.34.16_typescript#4.6.4/node_modules/vue-tsc
│ │ ├── package.json
│ │ ├── tsconfig.json
│ │ ├── tsconfig.node.json
│ │ └── vite.config.ts
│ ├── package-b
│ │ ├── index.ts
│ │ ├── node_modules
│ │ │ ├── .bin
│ │ │ │ ├── tsc
│ │ │ │ ├── tsserver
│ │ │ │ ├── vite
│ │ │ │ └── vue-tsc
│ │ │ ├── #myscope
│ │ │ │ └── core -> ../../../core
│ │ │ ├── typescript -> ../../../node_modules/.pnpm/typescript#4.6.4/node_modules/typescript
│ │ │ ├── vite -> ../../../node_modules/.pnpm/vite#2.9.9/node_modules/vite
│ │ │ ├── vite-plugin-vue2 -> ../../../node_modules/.pnpm/vite-plugin-vue2#2.0.1_dj3dtukdyynhbiqf2xbv2ocyei/node_modules/vite-plugin-vue2
│ │ │ ├── vue -> ../../../node_modules/.pnpm/vue#2.6.14/node_modules/vue
│ │ │ ├── vue-template-compiler -> ../../../node_modules/.pnpm/vue-template-compiler#2.6.14/node_modules/vue-template-compiler
│ │ │ └── vue-tsc -> ../../../node_modules/.pnpm/vue-tsc#0.34.16_typescript#4.6.4/node_modules/vue-tsc
│ │ ├── package.json
│ │ ├── tsconfig.json
│ │ ├── tsconfig.node.json
│ │ └── vite.config.ts
│ └── core
│ ├── index.ts
│ └── package.json
├── pnpm-lock.yaml
└── pnpm-workspace.yaml
In packages/package-a/package.json:
{
"name": "#myscope/package-a",
"version": "0.0.0",
"private": true,
"scripts": {
"build": "vue-tsc --noEmit && vite build"
},
"dependencies": {
"#myscope/core": "workspace:*",
"vue": "^3.2.25"
},
"devDependencies": {
"#vitejs/plugin-vue": "^2.3.3",
"typescript": "^4.5.4",
"vite": "^2.9.9",
"vue-tsc": "^0.34.7"
}
}
In packages/package-b/package.json
{
"name": "#myscope/package-b",
"version": "0.0.0",
"private": true,
"scripts": {
"build": "vue-tsc --noEmit && vite build"
},
"dependencies": {
"#myscope/core": "workspace:*",
"vue": "^2.6.14"
},
"devDependencies": {
"typescript": "^4.5.4",
"vite": "^2.9.9",
"vite-plugin-vue2": "^2.0.0",
"vue-template-compiler": "^2.6.14",
"vue-tsc": "^0.34.7"
}
}
When I run pnpm run build in package-b (Vue#2) I get the version mismatch error whereas it shouldnt:
> vue-tsc --noEmit && vite build
failed to load config from /path/to/my/project/monorepo/packages/package-b/vite.config.ts
error during build:
Error:
Vue packages version mismatch:
- vue#3.2.36 (/path/to/my/project/monorepo/node_modules/.pnpm/vue#3.2.36/node_modules/vue/index.js)
- vue-template-compiler#2.6.14 (/path/to/my/project/monorepo/node_modules/.pnpm/vue-template-compiler#2.6.14/node_modules/vue-template-compiler/package.json)
This may cause things to work incorrectly. Make sure to use the same version for both.
If you are using vue-loader#>=10.0, simply update vue-template-compiler.
If you are using vue-loader#<10.0 or vueify, re-installing vue-loader/vueify should bump vue-template-compiler to the latest.
at Object.<anonymous> (/path/to/my/project/monorepo/node_modules/.pnpm/vue-template-compiler#2.6.14/node_modules/vue-template-compiler/index.js:10:9)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object.<anonymous> (/path/to/my/project/monorepo/node_modules/.pnpm/vite-plugin-vue2#2.0.1_dj3dtukdyynhbiqf2xbv2ocyei/node_modules/vite-plugin-vue2/dist/utils/descriptorCache.js:34:42)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object.<anonymous> (/path/to/my/project/monorepo/node_modules/.pnpm/vite-plugin-vue2#2.0.1_dj3dtukdyynhbiqf2xbv2ocyei/node_modules/vite-plugin-vue2/dist/main.js:35:27)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
ELIFECYCLE Command failed with exit code 1.
ELIFECYCLE Command failed with exit code 1.
It seems that the resolved version doesn't follows package.json and uses Vue#3 instead of Vue#2. I don't understand why. Could this be a bug about how vue-template-compiler resolves Vue or something?
A "dirty" workaround I've found to get it work is to add hoist=false to .npmrc at the project's root but I don't know what are the possible side-effects I can get by doing this. Anyway, is there a cleaner way to fix this?
PS. This can be related to https://github.com/vuejs/vue/issues/11828 but I'm asking here do do not reply in a closed issue.
Answering my own question since I got help on the VueJS discord server.
There are two possible solutions that both worked for me independently:
a) move vue-tsc to the root package.json. This is what I did as I try and keep all my project tooling out of the individual packages.
b) use "pnpm.packageExtentions" in package.json to add vue#2 as a peer dependency of vue-template-compiler
Option a) doesn't seem to work as of pnpm#7 because of a breaking change:
Dependencies of the root workspace project are not used to resolve peer dependencies of other workspace projects #4469
Option B with pnpm.packageExtensions is working fine. No need to add nohoist=false anymore to .npmrc.
diff --git i/package.json w/package.json
index 95cda08..182e89d 100644
--- i/package.json
+++ w/package.json
## -41,5 +41,14 ##
"packageManager": "pnpm#7.1.3",
"engines": {
"node": ">=16.15.0"
+ },
+ "pnpm": {
+ "packageExtensions": {
+ "vue-template-compiler": {
+ "peerDependencies": {
+ "vue": "^2.6.14"
+ }
+ }
+ }
}
}
I want to add another page to vuepress with the blog plugin.
The new page that I added does not show the content.
I expect the about page to show the content
I use basic vuePressBlog template. My tree structure is
├── examples
│ ├── about
│ │ └── Readme.md
│ └── _posts
│ ├── 2018-4-4-intro-to-vuepress.md
│ ├── 2019-6-8-intro-to-vuepress-next.md
│ ├── 2019-6-8-shanghai.md
│ ├── 2019-6-8-summary.md
│ └── 2019-6-8-vueconf.md
├── index.js
├── layouts
│ ├── GlobalLayout.vue
│ ├── Layout.vue
│ ├── Post.vue
│ └── Tag.vue
├── package.json
├── package-lock.json
└── README.md
I added the following lines to the ./example/.vuepress/config.js
module.exports = {
title: "SlimBlog",
theme: require.resolve("../../"),
themeConfig: {
// Please keep looking down to see the available options.
nav: [
{
text: "Home",
link: "/",
},
{
text: "about",
link: "/about/",
},
{},
],
},
};
I assume there might be a layout missing but I unable to find the config for this.
I am struggling to work out how to pass outputs from a module and to consume it an another.
My folder structure:
.
├── main.tf
├── modules
│ ├── cloudwatch-event
│ │ ├── basic_event_rule.tf
│ │ ├── basic_event_target.tf
│ │ └── variables.tf
│ └── lambda
│ ├── basic_lambda.tf
│ ├── output.tf
│ ├── lambda.py
│ └── variables.tf
├── lambda
│ ├── main.tf
│ └── variables.tf
└── terraform.tfvar
In order to add scheduling to the lambda, i need to consume the Lambda ARN in to the CloudWatch module.
The lambda - basic_lambda.tf
resource "aws_lambda_function" "lambda_function" {
The lambda - outputs.tf
output "lambda_arn" {
value = "${aws_lambda_function.lambda_function.arn}"
In my lambda application module, i have this in my main lambda/main.tf
module "cloudwatch-event" {
source = "../modules/cloudwatch-event"
lambda_arn = "${module.lambda.lambda_arn}"
module "lambda" {
source = "../modules/lambda"
My lambda/variables.tf includes the lambda_arn variable as a string
variable "lambda_arn" {
type = "string"
}
The root main file looks like this:
provider "aws" {
region = var.aws_region
}
module "accesskey-lambda" {
source = "./lambda/"
}
Running TF i get this
Error: Missing required argument
on main.tf line 5, in module "accesskey-lambda":
5: module "accesskey-lambda" {
The argument "lambda_arn" is required, but no definition was found.
then adding it to the root main file doesnt resolve my issues.
Thanks
Nick
Solved, i had a typo
in the cloudwatch/basic_event_target.tf
arn = "${var.lambda_arn}"
Then in the cloudwatch/variable
variable "lambda_arn" {
type = string
}
The module then needed
module "cloudwatch-event" {
source = "../modules/cloudwatch-event"
lambda_arn = "${module.lambda.lambda_arn}"
}
I would like to extend my controller within simple widget.
I've created two files:
app/widgets/mywidget/controllers/base.js
app/widgets/mywidget/controllers/index.js
I start mycontroller.js file with line: exports.baseController = 'base'; and on Android it crashes with Exception:
/V8Exception(19693): Exception occurred at ti:/module.js:280: Uncaught Error: Requested module not found: alloy/controllers//glass/parent
Project tree looks like this:
app
├── README
├── alloy.js
├── assets
├── config.json
├── controllers
│ ├── base.js
│ ├── index.js
│ └── view.js
├── lib
│ └── user.js
├── models
├── styles
│ ├── app.tss
│ └── index.tss
├── views
│ ├── index.xml
│ └── view.xml
└── widgets
└── mywidget
├── controllers
│ ├── base.js
│ ├── index.js
│ └── view.js
├── styles
├── views
└── widget.json
index.js & view.js inside app/controller use base.js as baseController.
index.js & view.js inside app/widgets/mywidget/controllers use base.js inside same directory as their baseController. I don't try to extend baseController from app inside widget.