Scale Fargate service tasks to match CloudWatch metric - amazon-cloudwatch

I'm using CloudWatch Metric Math to calculate the number of workers (tasks) that I want my Fargate service to be scaled to. I planned on creating an alarm in CloudWatch to trigger the scaling action once it rose above or below the target number of 0. However, it doesn't look like there a way I can create an alarm based on CloudWatch Metric Math - or an alarm that does any type of comparison between two numbers (number of tasks needed vs. number of tasks existing).
How can I setup a Fargate scaling policy to scale based on my existing metric of 'Workers Needed'.
Metric Math
Formula: m1-m2-3 == desired scale offset
m1: Active Workers (tasks)
m2: Workers Needed (tasks)
{
"type": "metric",
"x": 0,
"y": 0,
"width": 24,
"height": 6,
"properties": {
"metrics": [
[ { "expression": "m1-m2-3", "label": "Workers/Needed difference", "id": "e1" } ],
[ "AWS/ECS", "MemoryUtilization", "ServiceName", "worker-service", "ClusterName", "my-cluster", { "period": 60, "stat": "SampleCount", "id": "m1", "label": "Active Workers" } ],
[ "LogMetrics", "Workers Needed", { "period": 60, "stat": "Maximum", "id": "m2" } ]
],
"view": "timeSeries",
"stacked": false,
"region": "us-east-1",
"title": "Worker/Lab difference",
"period": 300
}
}

Edit: Alarms based on metric math is now a thing
This doc page about metric math doesn't mention alarms at all, expressions seem to be more about visualizing with a dashboard. I also don't see anything about metric math in the SDK or cli documentation as far as alarms are concerned.
Your next simplest solution is probably paying homage to the great catch-all of all shortcomings of AWS, and write a Lambda that pulls the metrics, does the calculation, then publishes the metric as a custom metric with PutMetric. You can trigger this with a CloudWatch Event if you want a cron-like thing, or many, many other things by integrating it with SNS or just invoking it directly.
It's not the answer you want, but unfortunately I think it's the simplest way to get the functionality you want.

Related

JsonSchema number validation with multiple ranges

Is there a supported way (without using the anyOf keyword) to specify multiple ranges for a number in JsonSchema?
E.g., a value can either be in the range of 0-10 or 50-100 (the range of 10 < x < 50 is considered invalid).
The anyOf keyword can be used as follows:
{
"anyOf": [
{
"type": "number",
"minimum": 0,
"maximum": 10
},
{
"type": "number",
"minimum": 5,
"maximum": 100
}
]
}
Additionally, if the only allowed values were whole integers, I could use an enum and actually hand-specify each allowed number, but obviously that's less ideal than specifying ranges.
So, just wondering if there is a way to accomplish this with something like a "restrictions" keyword:
//NOTE: the below is not actually supported (I don't think), just using it as an example of what I'm interested in
{
"type": "number",
"restrictions": [
{
"minimum": 0,
"maximum": 10
},
{
"minimum": 50,
"maximum": 100
}
]
}
Also, for those wondering why if anyOf is available, it's that I have some custom tooling to maintain and supporting anyOf would be more of a lift than something that is specific to numeric validation.
The scenario you describe is exactly why anyOf exists. So no, if you want to express a logical OR, you need to implement the keyword that implements that. I don't see why adding a new custom keyword would make things any easier.

How do I get info about a Youtube video's chapters from the API?

Recently, Youtube added the ability to break up their videos in the progress bar into sections called "chapters".
https://support.google.com/youtube/answer/9884579?hl=en
Currently I am able to get info about a video from the Youtube API. However, it doesn't seem like there's any info about a video's chapters, and I haven't found anything in the API documentation about chapters. Am I missing something, or is there simply no way to get chapter data yet?
As far as I know, such data is in plain text in the description of the video.
So, you can use the following example:
Video used in this demonstration: Top 10 Monsters with 2500 Attack in YuGiOh
URL Request:
https://www.googleapis.com/youtube/v3/videos?part=snippet&id=NNgYId7b4j0&key=[YOUR_API_KEY]
Response:
{
"kind": "youtube#videoListResponse",
"etag": "YpVLmrSx1iP8hAJOnumaTBoKqqQ",
"items": [
{
"kind": "youtube#video",
"etag": "oIoJq5F3RHvBbtVohafaJ_1SThU",
"id": "NNgYId7b4j0",
"snippet": {
"publishedAt": "2020-09-14T18:37:46Z",
"channelId": "UC0roOaAn95Rtgoe078RkVXQ",
"title": "Top 10 Monsters with 2500 Attack in YuGiOh",
"description": "In this video we'll go over the best monsters that have 2500 attack, and attack threshold for a lot of boss monsters actually.\n\nCheck out my DnD channel #TheD&DLogs \n\n--The List--\nIntro: (0:00)\n10- Blue-Eyes Spirit Dragon: (0:00)\n9- Invoked Mechaba: (2:14)\n8- Number S39: Utopia the Lightning: (3:23)\n7- Earthbound Immortal Aslla Piscu: (4:35)\n6- Eldlich the golden Lord: (6:04)\n5- True King Lithosagym the Disaster: (7:34)\n4- Block Dragon: (8:54)\n3- Astrograph sorcerer: (10:25)\n2- Beatrice lady of the eternal: (12:36)\n1- Firewall Dragon: (14:37)\n- \n-----------------------------------------\n#yugioh #top10 \n\nDuels are all done on EDOpro, its completely free and updated all the time. If you want it, just look for the EDOpro discord and you'll find all you need to download it from there\n\nSome of the Video backgrounds in this video were made by \"Amitai Angor AA VFX\" https://www.youtube.com/dvdangor2011\n\n\nhttps://twitter.com/hirumared\nhttps://twitter.com/TheDuelLogs",
"thumbnails": {
"default": {
"url": "https://i.ytimg.com/vi/NNgYId7b4j0/default.jpg",
"width": 120,
"height": 90
},
"medium": {
"url": "https://i.ytimg.com/vi/NNgYId7b4j0/mqdefault.jpg",
"width": 320,
"height": 180
},
"high": {
"url": "https://i.ytimg.com/vi/NNgYId7b4j0/hqdefault.jpg",
"width": 480,
"height": 360
},
"standard": {
"url": "https://i.ytimg.com/vi/NNgYId7b4j0/sddefault.jpg",
"width": 640,
"height": 480
},
"maxres": {
"url": "https://i.ytimg.com/vi/NNgYId7b4j0/maxresdefault.jpg",
"width": 1280,
"height": 720
}
},
"channelTitle": "TheDuelLogs",
"tags": [
"yugioh",
"ygo",
"dev",
"pro",
"link",
"duels",
"auto-matic duels",
"online",
"current",
"ban",
"list",
"dueling",
"network",
"theduellogs",
"the",
"duel",
"logs",
"loggs",
"Yu",
"Gi",
"Oh!",
"YGOpro",
"gimmick",
"links",
"top ten",
"2020",
"edopro"
],
"categoryId": "20",
"liveBroadcastContent": "none",
"localized": {
"title": "Top 10 Monsters with 2500 Attack in YuGiOh",
"description": "In this video we'll go over the best monsters that have 2500 attack, and attack threshold for a lot of boss monsters actually.\n\nCheck out my DnD channel #TheD&DLogs \n\n--The List--\nIntro: (0:00)\n10- Blue-Eyes Spirit Dragon: (0:00)\n9- Invoked Mechaba: (2:14)\n8- Number S39: Utopia the Lightning: (3:23)\n7- Earthbound Immortal Aslla Piscu: (4:35)\n6- Eldlich the golden Lord: (6:04)\n5- True King Lithosagym the Disaster: (7:34)\n4- Block Dragon: (8:54)\n3- Astrograph sorcerer: (10:25)\n2- Beatrice lady of the eternal: (12:36)\n1- Firewall Dragon: (14:37)\n- \n-----------------------------------------\n#yugioh #top10 \n\nDuels are all done on EDOpro, its completely free and updated all the time. If you want it, just look for the EDOpro discord and you'll find all you need to download it from there\n\nSome of the Video backgrounds in this video were made by \"Amitai Angor AA VFX\" https://www.youtube.com/dvdangor2011\n\n\nhttps://twitter.com/hirumared\nhttps://twitter.com/TheDuelLogs"
},
"defaultAudioLanguage": "en"
}
}
],
"pageInfo": {
"totalResults": 1,
"resultsPerPage": 1
}
}
Get the response:
response.items[0].snippet.description
Results:
"In this video we'll go over the best monsters that have 2500 attack, and attack threshold for a lot of boss monsters actually.
Check out my DnD channel #TheD&DLogs
--The List--
Intro: (0:00)
10- Blue-Eyes Spirit Dragon: (0:00)
9- Invoked Mechaba: (2:14)
8- Number S39: Utopia the Lightning: (3:23)
7- Earthbound Immortal Aslla Piscu: (4:35)
6- Eldlich the golden Lord: (6:04)
5- True King Lithosagym the Disaster: (7:34)
4- Block Dragon: (8:54)
3- Astrograph sorcerer: (10:25)
2- Beatrice lady of the eternal: (12:36)
1- Firewall Dragon: (14:37)
-
-----------------------------------------
#yugioh #top10
Duels are all done on EDOpro, its completely free and updated all the time. If you want it, just look for the EDOpro discord and you'll find all you need to download it from there
Some of the Video backgrounds in this video were made by "Amitai Angor AA VFX" https://www.youtube.com/dvdangor2011
https://twitter.com/hirumared
https://twitter.com/TheDuelLogs"
One more time YouTube Data API v3 doesn't provide a basic feature.
I would suggest you to use my open-source YouTube operational API, indeed by requesting https://yt.lemnoslife.com/videos?part=chapters&id=VIDEO_ID you would get a JSON with the video chapters (titles and timestamps) you are looking for in item['chapters']['chapters'].
Example of result with YouTube video id NNgYId7b4j0:
{
"kind": "youtube#videoListResponse",
"etag": "NotImplemented",
"items": [
{
"kind": "youtube#video",
"etag": "NotImplemented",
"id": "NNgYId7b4j0",
"chapters": {
"areAutoGenerated": false,
"chapters": [
{
"title": "10- Blue-Eyes Spirit Dragon",
"time": 0,
"thumbnails": [
{
"url": "https:\/\/i.ytimg.com\/vi\/NNgYId7b4j0\/hqdefault_4000.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLCoTrvu0Yu-iNxb7o4II-pxi5WVbQ",
"width": 168,
"height": 94
},
{
"url": "https:\/\/i.ytimg.com\/vi\/NNgYId7b4j0\/hqdefault_4000.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLCuupNwIgFIf9hXbjMsvpSGThFyhg",
"width": 336,
"height": 188
}
]
},
{
"title": "9- Invoked Mechaba",
"time": 134,
"thumbnails": [
{
"url": "https:\/\/i.ytimg.com\/vi\/NNgYId7b4j0\/hqdefault_135933.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLBe94BKNpQXvM2dUl75LtcgX0N03w",
"width": 168,
"height": 94
},
{
"url": "https:\/\/i.ytimg.com\/vi\/NNgYId7b4j0\/hqdefault_135933.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLBULUhlI1OOjJiW6mpFDUhPzh4Adw",
"width": 336,
"height": 188
}
]
},
...
]
}
}
]
}
I am replying with this answer to help people such as myself who ended up on this video wanting to find a youtube chapter parser / extractor for text rather than where to find the chapter data. Just to add some further information, currently, there is no way to get the chapters from the official YouTube API, so the only way to get the chapters from a text-description response (like the YouTube API provides) is to parse it in some way:
My answer is in Javascript but it can easily be converted: The idea is to extract the MIN:SEC and HR:MIN:SEC timestamps then to generate the title we remove the word that includes them (So this would typically remove however people aesthetically wrap them too [00:00] or (00:00)
It's far from perfect, but in my experience it's better than the other solutions I've seen on github/npm at the time of writing this. You might want to also trim away starting and ending spaces and punctuational separators such as (-, :, ~, |) too
const parseChapters = (description) => {
// Extract timestamps (either 00:00:00, 0:00:00, 00:00 or 0:00)
const lines = description.split("\n")
const regex = /(\d{0,2}:?\d{1,2}:\d{2})/g
const chapters = []
for (const line of lines) {
// Match the regex and check if the line contains a matched regex
const matches = line.match(regex)
if (matches) {
const ts = matches[0]
const title = line
.split(" ")
.filter((l) => !l.includes(ts))
.join(" ")
chapters.push({
timestamp: ts,
title: title,
})
}
}
return chapters
}
Very late answer but it solved my problem.
You could use the code below. It's written in C# but it can easily be transcribed into another language. Since you can already get youtube video data, I assume you also have the description of the video.
private static IEnumerable<string> GetChaptersFromDescription(string text)
{
var lines = text.Split("\n");
var regex = new Regex(#"[0-9]:[0-9][0-9]");
foreach (var line in lines)
{
if (regex.IsMatch(line))
{
yield return line;
}
}
}

create order with different extra bag for outbound and inbound

I want to test order create API by adding extra bags. And I am experiencing a strange problem.
I make a search for Paris-NYC round trip, then I send the request to offer price API using include=detailed-fare-rules,bags parameter.
In the response, I get 2 kinds of extra bag information:
1 bag, 30 EUR
2 bags, 75 EUR
"bags": {
"1": {
"quantity": 1,
"name": "CHECKED_BAG",
"price": {
"amount": "30.00",
"currencyCode": "EUR"
},
"bookableByItinerary": true,
"segmentIds": [
"1",
"3"
],
"travelerIds": [
"1"
]
},
"2": {
"quantity": 2,
"name": "CHECKED_BAG",
"price": {
"amount": "75.00",
"currencyCode": "EUR"
},
"bookableByItinerary": true,
"segmentIds": [
"1",
"3"
],
"travelerIds": [
"1"
]
}
}
Everything goes well if I create order by:
adding 1 bag for outbound(paris to NYC), and adding 1 bag for inbound(NYC to Paris)
adding only 1 bag for outbound (0 extra bag for inbound)
adding 2 bag for outbound(paris to NYC), and adding 2 bags for inbound(NYC to Paris)
The problem is for the scenario:
I create order by adding 1 bag for outbound, and adding 2 bags for inbound.
In this case, the order is created with a warning message
"warnings": [
{
"status": 200,
"code": 0,
"title": "BookingWithPriceMarginWarning",
"detail": "The prices are lower than expected"
}
]
And the created order contains 1 extra bag for outbound, and 1 extra bag for inbound.
So I have 2 questions about this strange problem:
Is it normal that my order is modified when processing order create ?
Adding different number of extra bags for different itineraries is supported ?
Thanks
Is it normal that my order is modified when processing order create ?
It depends if you are Self-Service or Enterprise user:
For Enterprise users, Flight Create Orders offer the possibility to do a "best-effort" for additional-service booking. If this option is activated, Flight Create Orders gives priority to the reservation of your flight and remove the additional service that cannot be booked. That's why you receive the warning in your request when it happens.
Self Service users have the default behavior which rejects the creation of the order if at least one additional service can not be booked. In this case you will receive the following error:
{
"errors": [
{
"status": 400,
"code": 38034,
"title": "ONE OR MORE SERVICES ARE NOT AVAILABLE",
"detail": "Error booking additional services"
}
]
}
Adding different number of extra bags for different itineraries is supported ?
Yes, that is supported. Be aware that you cannot have an infinite amount of bag on a plane, so it could happen that you get an error when adding extra bags if there are too many bags already added by other passengers.

snap selection to week/month/year

I'm using Vega Lite in a crossfiltery app with an external data source.
I'd like the selection on a chart with a temporal scale to "snap" or "round" to a time interval which I'll supply in the spec.
I don't see anything about this in the selections documentation, so I guess I will probably need to generate a Vega spec from the Vega Lite spec, and then patch it. That's not something I've done yet, but I'm eager to learn.
However I am surprised not to find this question on SO, GitHub or Slack. I think I've searched for all combinations of {vega, vega-lite} x {round, snap}.
Closest I can find is an issue related to snapping selections for ordinal scales.
Am I using the wrong terminology?
It turns out the issue regarding snapping ordinal scales is more about snapping to time intervals, and this comment by Jeff Heer covers the topic thoroughly.
The resulting Vega expressions are lengthy, but the process is straightforward:
Get the x-coordinate value.
Run it through a scale inversion (x.invert) to map from pixel domain to data domain.
Perform "snapping" by rounding / truncating values in the data domain.
Re-apply the x scale to get the snapped pixel coordinate.
We can change the update fields of the signals to implement this truncation.
snap to year
Say the selection is named selection1. Vega-Lite will generate a Vega spec including the signal selection1_x (redacted for brevity):
{
"name": "selection1_x",
"value": [],
"on": [
{
"events": {
"source": "scope",
"type": "mousedown",
"filter": // ...
},
"update": "[x(unit), x(unit)]"
},
{
"events": {
"source": "scope",
"type": "mousemove",
"between": // ...
},
"update": "[selection1_x[0], clamp(x(unit), 0, width)]"
},
// ...
We can replace the x(unit) expressions in the first on item, and the clamp(x(unit), 0, width) expression in the second on item, using the steps above:
{
"name": "selection1_x",
"value": [],
"on": [
{
"events": {
"source": "scope",
"type": "mousedown",
"filter": // ...
},
"update": "[scale(\"x\",datetime(year(invert(\"x\",x(unit))),0)),scale(\"x\",datetime(year(invert(\"x\",x(unit))),0))]"
},
{
"events": {
"source": "scope",
"type": "mousemove",
"between": // ...
},
"update": "[selection1_x[0],scale(\"x\",datetime(year(invert(\"x\",clamp(x(unit),0,width))),0))]"
},
// ...
The lines are getting long, and this is without considering intervals smaller than a year. You probably don't want to write this stuff by hand!
smaller intervals
This method supports truncating to any of the parameters of the Vega datetime expression:
datetime(year, month[, day, hour, min, sec, millisec])
Returns a new Date instance. The month is 0-based, such that 1 represents February.
You have to specify at least year and month, so above we had zeros for month.
week
To snap to week, you can subtract the day of the week from the date of the month:
[selection1_x[0], scale("x", datetime(
year(invert("x", clamp(x(unit),0,width))),
month(invert("x", clamp(x(unit),0,width))),
date(invert("x", clamp(x(unit),0,width)))
- day(invert("x", clamp(x(unit),0,width)))))]
I haven't learned Vega well enough to know if I can simplify these expressions.
onward
Also vega-time is in the pipeline, providing the functionality of d3-time intervals, so this should get a lot simpler in the near future!

Get data Amadeus API Flight low fare search

I need parameters to get data that contains stops array. I tried about 100 different combinations, and i didn't get any response that returns stops array in results.
If anyone knows how to accomplish this, please provide your answer.
Thanks.
Having stops is not that common and it usually depends on the distance between origin and destination. For example, having London as origin and Sydney as destination:
https://test.api.amadeus.com/v1/shopping/flight-offers?origin=LON&destination=SYD&departureDate=2019-08-01&nonStop=false&returnDate=2019-08-28
You can check in the response that most of the segments contain stops:
"stops": [
{
"iataCode": "HKG",
"duration": "0DT1H0M",
"arrivalAt": "2019-08-28T12:00:00+08:00",
"departureAt": "2019-08-28T13:00:00+08:00"
},
{
"iataCode": "DOH",
"duration": "0DT1H0M",
"arrivalAt": "2019-08-28T14:00:00+03:00",
"departureAt": "2019-08-28T15:00:00+03:00"
},
{
"iataCode": "BAH",
"duration": "0DT1H0M",
"arrivalAt": "2019-08-28T16:00:00+03:00",
"departureAt": "2019-08-28T17:00:00+03:00"
}
]
Where stop means that an aircraft lands for refueling, for instance, but passengers don't necessary get out of the plane.