Replace newlines between two words - awk

I have a output from a text file as below. I want to put all the contents of the someItems array under one line. So, every line would have the contents of a new someItems array. For example :
"someItems": [
{
"someId": "MountSomers-showtime.com-ETTI0000000000000003-1452005472058",
"source": "MountSomers",
"sourceAssetId": "9",
"title": "Pk_3",
"ppp": "12",
"expirationDate": "2016-01-06T14:51:12Z"
}, {
"someId": "MountSomers-ericsson.com- ETTI0000000000000005-1452005472058",
"source": "MountSomers",
"sourceAssetId": "12",
"title": "Pk_5",
"ppp": "12",
"expirationDate": "2016-01-06T14:51:12Z"
} ]
"someItems": [
{
"someId": "MountSomers-hbo.com-ETTI0000000000000002-1452005472058",
"source": "MountSomers",
"sourceAssetId": "7",
"title": "Pk_2",
"ppp": "12",
"expirationDate": "2016-01-06T14:51:12Z"
}, {
"someId": "MountSomers-showtime.com-ETTI0000000000000003-1452005472058",
"source": "MountSomers",
"sourceAssetId": "9",
"title": "Pk_3",
"ppp": "12",
"expirationDate": "2016-01-06T14:51:12Z"
}, {
"someId": "MountSomers-ericsson.com-ETTI0000000000000005-1452005472058",
"source": "MountSomers",
"sourceAssetId": "12",
"title": "Pk_5",
"ppp": "12",
"expirationDate": "2016-01-06T14:51:12Z"
} ]
would become
"someItems": [ ..... ]
"someItems": [ ..... ]
I have the below
cat file | | awk '/^"someItems": [/{p=1}/^]/{p=0} {if(p)printf "%s",$0;else printf "%s%s\n",(NR==1?"":RS),$0}'
but it does not do what I wanted...

Since the input contains the brackets [] only in the outer level the solution can be pretty simple:
awk '{gsub("\n","", $0)}1' RS=']\n' file
I'm using ]\n as the input record separator. This gives you the whole portion between "someItems: ..." until the closing ] as $0. gsub() simply replaces the newlines. 1 prints the (modified) record.
You can also use sed:
sed '/\[/{:a;N;/]/!ba;s/\n//g}' file
I'll explain it in a multiline version:
script.sed:
# Address. Matches a line containing the opening [
/\[/ { # Start of block
# Define a label 'a'
:a
# Read a new line and append it to the pattern buffer
N
# If the pattern buffer doens't contain the closing ]
# jump back to label 'a'
/]/!ba
# Replace all newlines once the closing bracket appeared
# Since we don't jump back to 'a' in this case, this means we'll
# leave the block and start a new cycle.
s/\n//g
} # End of block

$ awk '/^"someItems":/ && f { printf "\n" } { printf $0; f=1 } END { printf "\n" }' file.txt
"someItems": [{ "someId": "MountSomers-showtime.com-ETTI0000000000000003-1452005472058", "source": "MountSomers", "sourceAssetId": "9", "title": "Pk_3", "ppp": "12", "expirationDate": "2016-01-06T14:51:12Z"}, { "someId": "MountSomers-ericsson.com- ETTI0000000000000005-1452005472058", "source": "MountSomers", "sourceAssetId": "12", "title": "Pk_5", "ppp": "12", "expirationDate": "2016-01-06T14:51:12Z"} ]
"someItems": [{ "someId": "MountSomers-hbo.com-ETTI0000000000000002-1452005472058", "source": "MountSomers", "sourceAssetId": "7", "title": "Pk_2", "ppp": "12", "expirationDate": "2016-01-06T14:51:12Z"}, { "someId": "MountSomers-showtime.com-ETTI0000000000000003-1452005472058", "source": "MountSomers", "sourceAssetId": "9", "title": "Pk_3", "ppp": "12", "expirationDate": "2016-01-06T14:51:12Z"}, { "someId": "MountSomers-ericsson.com-ETTI0000000000000005-1452005472058", "source": "MountSomers", "sourceAssetId": "12", "title": "Pk_5", "ppp": "12", "expirationDate": "2016-01-06T14:51:12Z"} ]
$
Print each line without a trailing newline. Starting with the second occurrence, put a leading newline before each "someItems". Print a newline at the end to keep it classy.

Related

removing pattern from lines in a file using sed or awk

I am trying to remove a pattern from all lines in a file. The pattern is 'id': null and the two sed attempts I have made execute but the file is unchanged.. Thank you :).
file
{
"objects": [
{
"version": "1a",
"chromosome": "chr1",
"id": null,
"peptide": "123",
},
{
"version": "1a",
"chromosome": "chr1",
"id": "This line has text and is printed.",
"peptide": null,
},
{
"version": '1a',
"chromosome": "chr17",
"id": null,
"peptide": null},
"id": 'This has text in it and this line is printed as well',
"end": 460
}
]
}
desired
{
"objects": [
{
"version": "1a",
"chromosome": "chr1",
"peptide": "123",
},
{
"version": "1a",
"chromosome": "chr1",
"id": "This line has text and is printed.",
"peptide": null,
},
{
"version": '1a',
"chrmosome": "chr17",
"id": null,
"peptide": null},
"id": 'This has text in it and this line is printed as well',
"end": 460
}
]
}
sed
sed '/"id": *null/s/, [^,]*/ /' file --- if "id": null found substitute with blank up to the ending ,
sed -E "s/"id": *null, *//" file
You may use this gnu-sed:
sed 0,'/"id": null,/{//d}' file
This will remove first instance of "id": null, from file
Original answer based on original question:
sed -E "s/'id': *None, *//" file
{'version': '1a', 'chr': '17', 'xref_json': None}, 'id': 'This has text in it and this line is printed', 'end': 460}
{'version': '1a', 'chr': '17', 'xref_json': None}
s/'id': *None, *// searches for pattern 'id':<0 or more spaces>None,<0 or more spaces> and replaces that with empty string.

Setting up Continuous export of API data to csv for Racing lap times

Have access to an API that provides data for racing, including driver names, and their last lap, best lap.. etc. Completely new to coding but learning the ropes. This is an example of an output from the API.
"Successful": true,
"Session": {
"RunNumber": "47",
"SessionName": "KART DRIVERS - EXPERIENCE / PROVA 11 22:30",
"TrackName": "KGV RACE TRACKS - CIRCUITO 109",
"TrackLength": "0.725",
"CurrentTime": "23:05:24",
"SessionTime": "00:13:27",
"TimeToGo": "00:04:32",
"LapsToGo": "9999",
"FlagStatus": "Green",
"SortMode": "race",
"Classes": {
"1": {
"ClassID": "1",
"Description": "RENTAL"
}
},
"Competitors": {
"018": {
"RacerID": "018",
"Number": "018",
"Transponder": "02",
"FirstName": "LR",
"LastName": "",
"Nationality": "",
"AdditionalData": "",
"ClassID": "1",
"Position": "28",
"Laps": "8",
"TotalTime": "00:12:34.376",
"BestPosition": "26",
"BestLap": "8",
"BestLapTime": "00:01:09.158",
"LastLapTime": "00:01:09.158"
},
"043": {
"RacerID": "043",
"Number": "043",
"Transponder": "48",
"FirstName": "LORENZO",
"LastName": "",
"Nationality": "",
"AdditionalData": "",
"ClassID": "1",
"Position": "32",
"Laps": "5",
"TotalTime": "00:12:54.095",
"BestPosition": "32",
"BestLap": "4",
"BestLapTime": "00:01:38.740",
"LastLapTime": "00:02:39.277"
How would I go about reading data from this api every 30 seconds to 1 minute, and exporting that data into a CSV to put on excel?

TextMate grammar - clashing multi-line captures

I have the following syntax:
test: name {
param_name: value
another_param: value2
test: [12, "asd"]
test2: [
"test__",
"test3"
]
}
My logic here is as follows:
Detect scopes as multi-line match
"begin": "([a-z_]+)\\s?:\\s?([a-z_\\+]+)?\\s?(\\{)",
"end": "(\\})",
In the patterns section of the above, add parameters with multiline matching
"begin": "(?!sql)([a-z\\_]*)\\s?:",
"end": "(?<=\\n)",
And then in the patterns of that I have array
"begin": "\\[",
"end": "\\]",
The problem is that test: [12, "asd"] is correctly defined as
test - parameter name
[12, "asd"] - parameter value + array
but I can't get it to work on the multi-line value. It only recognises the opening [ as array.
At first I thought I understood the reason why. The parameters finishes when it sees a new line, hence the second line of an array will not be matched. So I added array to the main scope pattern and that's when my understanding ends.
Full file:
{
"$schema": "https://raw.githubusercontent.com/martinring/tmlanguage/master/tmlanguage.json",
"name": "QQQL",
"patterns": [
{"include": "#scopes"},
{"include": "#parameters"}
],
"repository": {
"scopes": {
"name": "source.qqql.scope",
"begin": "([a-z_]+)\\s?:\\s?([a-z_\\+]+)?\\s?(\\{)",
"end": "(\\})",
"patterns": [
{"include": "#scopes"},
{"include": "#array"},
{"include": "#parameters"}
]
},
"parameters": {
"name": "source.qqql.parameter",
"begin": "(?!sql)([a-z\\_]*)\\s?:",
"end": "(?<=\\n)",
"beginCaptures": {
"1": {
"name": "source.qqql.parameter.name"
}
},
"patterns": [
{"include": "#array"},
{
"name": "source.qqql.parameter.value",
"match": "(.*)",
"captures": {
"1": {
"patterns": [
{"include": "#array"}
]
}
}
}
]
},
"array": {
"name": "source.qqql.array",
"begin": "\\[",
"end": "\\]",
"patterns": [
{"include": "#strings"},
{
"name": "source.qqql.array.delimiter",
"match": "\\,"
}
]
}
},
"scopeName": "source.qqql"
}
What I expected is that the inclusion of array in scopes would solve the problem but somehow it doesn't.

How to iterate and render my fetch result in react native JSX

{
"status": true,
"live_score_domestic": [
{
"nTournamentID": "1",
"cTournamentName": "sample tournament.",
"cTournamentType": "D",
"dStartDate": "2016-12-10",
"dEndDate": "2016-12-12",
"matches": [
{
"cVenueCode": "TTAB",
"cTableName": "Table 1",
"cEventType": "Junior Boys",
"cMatchNo": "5",
"cRound": "First Round",
"nScheduledDate": "2016-12-11",
"nScheduledTime": "11:45:00",
"teamname1": "MOTHER SCHOOL",
"teamname2": "HARI SHEWA SCHOOL",
"nVenueID": "1",
"nTableID": "1",
"nTeamID1": "3",
"nTeamID2": "4",
"nTournamentID": "1",
"nFixtureDetailsID": "15",
"nEventTypeID": "5",
"image": "http://example.com/tt.png"
},
{
"cVenueCode": "TTAB",
"cTableName": "Table 1",
"cEventType": "Junior Boys",
"cMatchNo": "4",
"cRound": "First Round",
"nScheduledDate": "2016-12-11",
"nScheduledTime": "11:30:00",
"teamname1": "MOTHER SCHOOL",
"teamname2": "HARI SHEWA SCHOOL",
"nVenueID": "1",
"nTableID": "1",
"nTeamID1": "3",
"nTeamID2": "4",
"nTournamentID": "1",
"nFixtureDetailsID": "14",
"nEventTypeID": "5",
"image": "http://example.com/tt.png"
}
]
}
],
"live_score_international": [
{
"nTournamentID": "2",
"cTournamentName": "International Tournament Sample",
"cTournamentType": "I",
"dStartDate": "2016-12-22",
"dEndDate": "2016-12-24",
"matches": []
}
],
"results_domestic": [
{
"nTournamentID": "1",
"cTournamentName": "sample tournament.",
"cTournamentType": "D",
"dStartDate": "2016-12-10",
"dEndDate": "2016-12-12",
"matches": [
{
"cVenueCode": "TTAB",
"cTableName": "Table 1",
"cEventType": "Junior Boys",
"cMatchNo": "5",
"cRound": "First Round",
"nScheduledDate": "2016-12-11",
"nScheduledTime": "11:45:00",
"teamname1": "MOTHER SCHOOL",
"teamname2": "HARI SHEWA SCHOOL",
"nVenueID": "1",
"nTableID": "1",
"nTeamID1": "3",
"nTeamID2": "4",
"nTournamentID": "1",
"nFixtureDetailsID": "15",
"nEventTypeID": "5",
"image": "http://example.com/tt.png"
},
{
"cVenueCode": "TTAB",
"cTableName": "Table 1",
"cEventType": "Junior Boys",
"cMatchNo": "4",
"cRound": "First Round",
"nScheduledDate": "2016-12-11",
"nScheduledTime": "11:30:00",
"teamname1": "MOTHER SCHOOL",
"teamname2": "HARI SHEWA SCHOOL",
"nVenueID": "1",
"nTableID": "1",
"nTeamID1": "3",
"nTeamID2": "4",
"nTournamentID": "1",
"nFixtureDetailsID": "14",
"nEventTypeID": "5",
"image": "http://example.com/tt.png"
}
]
}
],
"results_international": [
{
"nTournamentID": "2",
"cTournamentName": "International Tournament Sample",
"cTournamentType": "I",
"dStartDate": "2016-12-22",
"dEndDate": "2016-12-24",
"matches": []
}
],
"fixture_point_domestic": [
{
"nTournamentID": "1",
"cTournamentName": "sample tournament.",
"cTournamentType": "D",
"dStartDate": "2016-12-10",
"dEndDate": "2016-12-12"
}
],
"fixture_point_international": [
{
"nTournamentID": "2",
"cTournamentName": "International Tournament Sample",
"cTournamentType": "I",
"dStartDate": "2016-12-22",
"dEndDate": "2016-12-24"
}
]
}
This is the fetch result.I want to render first a heading like domestic Tournaments.Then i want to loop for al tornaments in domestic.Then all matches in each tournament.How i can do this?Anyone to help.thanks in advance :)
can you please help me to figure out how to iterate all
First of all you should decide on how the data should be displayed on the screen. Should they be clickable or not, scrolled or not, what part of screen they should take etc. And depending on the visual design of your future application you can make a choice what exact react-native visual component will represent the data on the screen by the best way.
For example, if it is applicable by visual design of your application you can take a look at the react-native ListView or ScrollView components to render endless list of identical complex data.
If iterating through that entire JSON object is your intent, then an easy approach is a for...in loop.
For example:
var obj = {a: 1, b: 2, c: {a: 1, b: 2}};
function walk(obj) {
for (var key in obj) {
if (obj.hasOwnProperty(key)) {
var val = obj[key];
console.log(val);
walk(val);
}
}
}
walk(obj);
Taken from: iterating through json object javascript
Object.keys(data).map((key) => { ... })
This will use the object properties (live_score_domestic, live_score_international, etc) to iterate through the top most data. Then you can use data[key] to get throught its contents.

how get multiple line starting with and format them

I Have a file which contains multiple lines.I want only few things from that output.Below is the output i am getting from server.
Output:
"az": "nova",
"cloud": "envvars",
"config_drive": "",
"created": "2016-08-19T17:21:24Z",
"flavor": {
"id": "4",
"name": "m1.large"
},
"hostId": "f714baee5967dc17e7d36c7b72eb92a4f1ab68d9782fa90a968ceae5",
"human_id": "dsc-test-3",
"id": "3f0a1188-c151-4e5e-9930-969d0423601b",
"image": {
"id": "7f4ad1f4-6fab-4978-b65a-ec4b9a407c5c",
"name": "mitel-dsc-7.9-9.15_3nic"
},
"interface_ip": "172.16.17.15",
"key_name": "key1",
"metadata": {},
"name": "dsc-test-3",
"networks": {
"dsc-InterInstance": [
"172.16.18.15"
],
"dsc-OAM": [
"172.16.16.20"
],
"dsc-sig": [
"172.16.17.15",
"10.10.72.15"
]
},
My intention is to get below things
Required:
"networks": {
"dsc-InterInstance": [
"172.16.18.15"
],
"dsc-OAM": [
"172.16.16.20"
],
"dsc-sig": [
"172.16.17.15",
"10.10.72.15"
]
}
Try this method
sed -n '/networks.*{/,/}/p' fileName
Outputs:
"networks": {
"dsc-InterInstance": [
"172.16.18.15"
],
"dsc-OAM": [
"172.16.16.20"
],
"dsc-sig": [
"172.16.17.15",
"10.10.72.15"
]
}
Awk solution:
awk '/"networks"/,/}/' file.txt
This gives the output
"networks": {
"dsc-InterInstance": [
"172.16.18.15"
],
"dsc-OAM": [
"172.16.16.20"
],
"dsc-sig": [
"172.16.17.15",
"10.10.72.15"
]
},
The syntax used here is /start_regex/,/stop_regex/, which starts matching when a line matches start_regex and stops matching after a line matches stop_regex (so all the lines in between also get matched). Since no action is specified, the default {print} action is used.
Compared to your requirement, this outputs an extra , on the last line, since that comma is present in the input. If that is unacceptable, you could get rid of it using the action {sub("},","}");print}. Or by using sed as in the other answer.
... and the tagged grep solution as well:
$ cat file|tr '\n' _| grep -o \[^_\]*\"networks\"\[^}\]*}|tr _ '\n'
"networks": {
"dsc-InterInstance": [
"172.16.18.15"
],
"dsc-OAM": [
"172.16.16.20"
],
"dsc-sig": [
"172.16.17.15",
"10.10.72.15"
]
}
ie. change every \n to _, grep out the requested block and put the \n back. _ may not be the best choice for a substitute but works in this particular case.
With GNU awk for multi-char RS and RT you can just use a regexp to describe the string you're looking for:
$ awk -v RS='"networks": {[^}]+}' 'RT{print RT}' file
"networks": {
"dsc-InterInstance": [
"172.16.18.15"
],
"dsc-OAM": [
"172.16.16.20"
],
"dsc-sig": [
"172.16.17.15",
"10.10.72.15"
]
}