Related
Let's say I want to predict the winner of a tag-team race, where some drivers are more usually place higher in certain weather conditions:
Race |Driver | Weather | Time
Dummy1 |D1 | Rain | 2:00
Dummy1 |D2 | Rain | 5:00
Dummy1 |D3 | Rain | 4:50
Dummy2 |D1 | Sunny | 3:00
Dummy2 |D2 | Sunny | 2:50
Dummy2 |D2 | Sunny | 2:30
...
The logic is that a team composed of D1 and D3 would outperform any other combination on Rain, but wouldn't have the same luck on other weather. With that said, I thought about the following model:
Layer 1 | Layer 2 | Layer 3 (output)
Driver encoding | weather encoding | expected race time
----------------------------------------------------------------
Input of 0 or 1 | sum(Layer 1 * weights | sum(Layer 2 * weights)
| * Input of 0 or 1) |
This means that layer 2 uses layer 1 as well as input values to compute a value.
The reason I want this architecture instead of having every feature on layer 1 is that I want different features to multiply each other instead of their sum.
I could not find anything like this, but it is probably just me not knowing the name of this approach. Can someone point me to sources or explain know how to replicate this on tensorflow/pytorch/any other lib?
Turns out it was actually pretty simple, for anyone that might stumble upon this post and would like to test this approach, here's rough code:
Racing dataset
# TEAM 1 TEAM 2 "Weather" "WON"
# "A","B","C","D","E", "A","B","C","D","E", W1 W2 W3 (combined times of team 1< combined times of team 2)
dataset=[[ 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1],
[ 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1],
[ 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1],
[ 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1],
[ 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0],
[ 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0],
[ 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0],
[ 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0],
[ 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0],
[ 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1],
]
inputs=[[x[0:-4],x[-4:-1]] for x in dataset]
results=[[x[-1]] for x in dataset]
Typings to make code more readable
from typing import Iterator
class InputLayer():
def __init__(self, inputs,useBias=False):
self.inputs=inputs
self.useBias=useBias
def __str__(self):
return "Layer of size "+ str(self.inputs)
def __repr__(self) -> str:
return self.__str__()
class InputLayerValue():
def __init__(self, values):
self.values=np.array(values)
Actual model
import torch
from torch import nn
class MutipleInputModel(nn.Module):
def __init__(self,input_layers:Iterator[InputLayer],output_size):
super(MutipleInputModel, self).__init__()
print(input_layers)
self.nns=[]
for i in range(len(input_layers)-1):
current:InputLayer=input_layers[i]
next:InputLayer=input_layers[i+1]
il=nn.Linear(current.inputs,next.inputs,current.useBias)
#To have hidden layers, you need to either use another model or create and attach multiple Linear models - nn.Linear(next.inputs,next.inputs)
name="nn"+str(i)
#models must be directly under self to be found by model.parameters()
self.__setattr__(name,il)
self.nns.append(name)
il=nn.Linear(input_layers[-1].inputs,output_size,current.useBias)
name="nnOutput"
self.__setattr__(name,il)
self.nns.append(name)
def forward(self, inputs:Iterator[InputLayerValue]):
inputsLen=len(inputs[0])
if inputsLen != len(self.nns):
raise Exception("Number of input values provided and input layers must be equal. Provided "+str(inputsLen)+" sets of inputs for a "+str(len(self.nns))+"-input-layer network")
#Initialize first layer of inputs with ones which will then be multiplied by the actual input values
lastOutput=torch.ones(len(inputs),len(inputs[0][0].values)) # Layer 1 Outputs | Layer 2 provided Inputs | Layer 2 actual Inputs
for i in range(inputsLen): # lastOutput | multiplier | input
multiplier=torch.from_numpy(np.array([x[i].values for x in inputs])).float() # 0.2 | 0 | 0
input=lastOutput*multiplier # 1.5 | 1 | 1.5
lastOutput=self.__getattr__(self.nns[i])(input) # 1.0 | 5 | 5
return lastOutput
Training
# Define hyperparameters
model = MutipleInputModel(input_layers=[InputLayer(len(x)) for x in inputs[0]],output_size=1)
n_epochs = 1000
lr=0.001
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
for epoch in range(1, n_epochs + 1):
optimizer.zero_grad() # Clears existing gradients from previous epoch
output = model([[InputLayerValue(y) for y in x] for x in inputs])
loss = criterion(output, torch.from_numpy(np.array(results)).float())
loss.backward()
optimizer.step()
print('Epoch: {}/{}.............'.format(epoch, n_epochs), end=' ')
print("Loss: {:.4f}".format(loss.item()))
Testing:
def predict(model, input):
input = [[InputLayerValue(y) for y in input]]
out = model(input)
return nn.Sigmoid()(out[0][0]).item()
print(predict(model,[[1, 1, 0, 0, 0, 0, 0, 1, 1, 0], [1, 0, 0]]))
print(predict(model,[[1, 1, 0, 0, 0, 0, 0, 1, 1, 0], [0, 1, 0]]))
print(predict(model,[[1, 1, 0, 0, 0, 0, 0, 1, 1, 0], [0, 0, 1]]))
This is a really basic implementation, but could easily be modified to have hidden layers.
Clearly needs further testing to see if it is actually better than a traditional NN, but I would say it is great for NN explainability.
I ran a LIWC analysis and it gives me the following results (below). I would like to turn the result into a pandas dataframe. If anyone can chip in, that would be wonderful.
Thanks in advance :)
Best,
David
resp = requests.post(url, auth=(api_key, api_secret), data=data)
resp1 = resp
print(resp.json())
{'plan_usage': {'call_limit': 1000, 'calls_made': 6, 'calls_remaining': 994, 'percent_used': 0.6, 'start_date': '2020-12-09T03:05:57.779556Z', 'end_date': '2020-12-23T03:05:57.779556Z'}, 'results': [{'response_id': 'd1382f42-5c28-4528-ab2e-81b80ba185e2', 'request_id': 'req-1', 'language': 'en', 'version': 'v1.0.0', 'summary': {'word_count': 57, 'words_per_sentence': 11.4, 'sentence_count': 5, 'six_plus_words': 0.2982456140350877, 'emojis': 0, 'emoticons': 0, 'hashtags': 0, 'urls': 0}, 'liwc': {'scores': {'analytical_thinking': 80.77394876079086, 'authentic': 38.8220872694557, 'clout': 50, 'emotional_tone': 97.58138119866139, 'dictionary_words': 0.8771929824561403, 'categories': {'achievement': 0, 'adjectives': 0.017543859649122806, 'adverbs': 0.03508771929824561, 'affect': 0.05263157894736842, 'affiliation': 0.017543859649122806, 'all_punctuation': 0.10526315789473684, 'anger_words': 0, 'anxiety_words': 0, 'apostrophes': 0, 'articles': 0.12280701754385964, 'assent': 0, 'auxiliary_verbs': 0.14035087719298245, 'biological_processes': 0, 'body': 0, 'causation': 0, 'certainty': 0, 'cognitive_processes': 0.05263157894736842, 'colons': 0, 'commas': 0.017543859649122806, 'comparisons': 0, 'conjunctions': 0.07017543859649122, 'dashes': 0, 'death': 0, 'differentiation': 0, 'discrepancies': 0.017543859649122806, 'drives': 0.03508771929824561, 'exclamations': 0, 'family': 0, 'feel': 0, 'female': 0, 'filler_words': 0, 'focus_future': 0, 'focus_past': 0, 'focus_present': 0.14035087719298245, 'friends': 0.017543859649122806, 'function_words': 0.543859649122807, 'health': 0, 'hear': 0, 'home': 0, 'i': 0.03508771929824561, 'impersonal_pronouns': 0.03508771929824561, 'informal_language': 0, 'ingestion': 0, 'insight': 0, 'interrogatives': 0.017543859649122806, 'leisure': 0.14035087719298245, 'male': 0, 'money': 0, 'motion': 0.05263157894736842, 'negations': 0, 'negative_emotion_words': 0, 'netspeak': 0, 'nonfluencies': 0, 'numbers': 0, 'other_grammar': 0.2807017543859649, 'other_punctuation': 0, 'parentheses': 0, 'perceptual_processes': 0.017543859649122806, 'periods': 0.08771929824561403, 'personal_concerns': 0.14035087719298245, 'personal_pronouns': 0.03508771929824561, 'positive_emotion_words': 0.05263157894736842, 'power': 0, 'prepositions': 0.10526315789473684, 'pronouns': 0.07017543859649122, 'quantifiers': 0.05263157894736842, 'question_marks': 0, 'quotes': 0, 'relativity': 0.17543859649122806, 'religion': 0, 'reward': 0.017543859649122806, 'risk': 0, 'sad_words': 0, 'see': 0.017543859649122806, 'semicolons': 0, 'sexual': 0, 'she_he': 0, 'social': 0.03508771929824561, 'space': 0.10526315789473684, 'swear_words': 0, 'tentative': 0.03508771929824561, 'they': 0, 'time': 0.017543859649122806, 'time_orientation': 0.14035087719298245, 'verbs': 0.19298245614035087, 'we': 0, 'work': 0, 'you': 0}}}, 'sallee': {'counts': {'emotions': {'admiration': 5, 'amusement': 0, 'anger': 0, 'boredom': 0, 'calmness': 0, 'curiosity': 0, 'desire': 0, 'disgust': 0, 'excitement': 0.375, 'fear': 0, 'gratitude': 2, 'joy': 6.375, 'love': 5, 'pain': 0, 'sadness': 0, 'surprise': 0}, 'goodfeel': 13.375, 'ambifeel': 0, 'badfeel': 0, 'emotionality': 13.375, 'sentiment': 13.375, 'non_emotion': None}, 'scores': {'emotions': {'admiration': 0.3333333333333333, 'amusement': 0, 'anger': 0, 'boredom': 0, 'calmness': 0, 'curiosity': 0, 'desire': 0, 'disgust': 0, 'excitement': 0.03614457831325301, 'fear': 0, 'gratitude': 0.16666666666666666, 'joy': 0.3893129770992366, 'love': 0.3333333333333333, 'pain': 0, 'sadness': 0, 'surprise': 0}, 'goodfeel': 0.2015065913370998, 'ambifeel': 0, 'badfeel': 0, 'emotionality': 0.2015065913370998, 'sentiment': 0.6541600137038615, 'non_emotion': 0.7984934086629002}, 'emotion_word_count': 4}}]}
js = resp.json()
df = pd.json_normalize(js['results'][0])
df.columns
Index(['response_id', 'request_id', 'language', 'version',
'summary.word_count', 'summary.words_per_sentence',
'summary.sentence_count', 'summary.six_plus_words', 'summary.emojis',
'summary.emoticons',
...
'sallee.scores.emotions.pain', 'sallee.scores.emotions.sadness',
'sallee.scores.emotions.surprise', 'sallee.scores.goodfeel',
'sallee.scores.ambifeel', 'sallee.scores.badfeel',
'sallee.scores.emotionality', 'sallee.scores.sentiment',
'sallee.scores.non_emotion', 'sallee.emotion_word_count'],
dtype='object', length=150)
df.iloc[0]
response_id d1382f42-5c28-4528-ab2e-81b80ba185e2
request_id req-1
language en
version v1.0.0
summary.word_count 57
...
sallee.scores.badfeel 0
sallee.scores.emotionality 0.202
sallee.scores.sentiment 0.654
sallee.scores.non_emotion 0.798
sallee.emotion_word_count 4
Name: 0, Length: 150, dtype: object
Problem: how to write a sqlite statement to select a value from a nested json object when the needed name is dynamic / variable. It is also important that this can be done from a single sql statement. Eventually, this will be executed from within a bash script.
In the object sample below, I need to list all the dot11.advertisedssid.ssid in the sql database. An acceptable solution is to list all values of dot11.advertisedssid.ssid that exist in the json object, but I would like to understand how to query a dynamic json name (so I can get the other nested values).
In general I am using json_extract in my sql statement I just can’t figure out how to get to the ssid value (in this example)!
How do I know 733545801 is the field name and how can I then use it in the json_extract statement? And do that for all such nested objects.
Examples:
In general this is how I am querying other json values.
select json_extract(devices.device,'$."dot11.device"."dot11.device.typeset"') from devices;
An object sample from the database:
"dot11.device": {
"dot11.device.typeset": 257,
"dot11.device.client_map": {
},
"dot11.device.num_client_aps": 0,
"dot11.device.advertised_ssid_map": {
"733545801": {
"dot11.advertisedssid.ssid": "SampleFES-WiFi",
"dot11.advertisedssid.ssidlen": 15,
"dot11.advertisedssid.beacon": 1,
"dot11.advertisedssid.probe_response": 1,
"dot11.advertisedssid.channel": "6",
"dot11.advertisedssid.ht_mode": "HT20",
"dot11.advertisedssid.ht_center_1": 0,
"dot11.advertisedssid.ht_center_2": 0,
"dot11.advertisedssid.first_time": 1559567379,
"dot11.advertisedssid.last_time": 1559567379,
"dot11.advertisedssid.beacon_info": "",
"dot11.advertisedssid.cloaked": 0,
"dot11.advertisedssid.crypt_set": 268436162,
"dot11.advertisedssid.maxrate": 65.000000,
"dot11.advertisedssid.beaconrate": 10,
"dot11.advertisedssid.beacons_sec": 2,
"dot11.advertisedssid.ietag_checksum": 1220416683,
"dot11.advertisedssid.wpa_mfp_required": 0,
"dot11.advertisedssid.wpa_mfp_supported": 0,
"dot11.advertisedssid.dot11d_country": "",
"dot11.advertisedssid.dot11d_list": [
],
"dot11.advertisedssid.wps_state": 0,
"dot11.advertisedssid.dot11r_mobility": 0,
"dot11.advertisedssid.dot11r_mobility_domain_id": 0,
"dot11.advertisedssid.dot11e_qbss": 0,
"dot11.advertisedssid.dot11e_qbss_stations": 0,
"dot11.advertisedssid.dot11e_channel_utilization_perc": 0.000000,
"dot11.advertisedssid.ccx_txpower": 0,
"dot11.advertisedssid.cisco_client_mfp": 0,
"dot11.advertisedssid.ie_tag_list": [
0.000000,
1.000000,
3.000000,
5.000000,
42.000000,
50.000000,
48.000000,
45.000000,
61.000000,
127.000000,
221.000000
]
}
}
Thanks for the help!
PS. This is from the new kismet database and the redesigned schema.
Here is the whole object:
{
"kismet.device.base.manuf": "Texas Instruments",
"kismet.device.base.key": "4202770D00000000_AFB4F569D2380000",
"kismet.device.base.macaddr": "38:D2:69:F5:B4:AF",
"kismet.device.base.phyname": "IEEE802.11",
"kismet.device.base.phyid": 0,
"kismet.device.base.name": "LincolnFES-WiFi",
"kismet.device.base.commonname": "LincolnFES-WiFi",
"kismet.device.base.type": "Wi-Fi AP",
"kismet.device.base.basic_type_set": 1,
"kismet.device.base.crypt": "WPA2-PSK",
"kismet.device.base.basic_crypt_set": 2,
"kismet.device.base.first_time": 1559567379,
"kismet.device.base.last_time": 1559567379,
"kismet.device.base.mod_time": 1559567380,
"kismet.device.base.packets.total": 3,
"kismet.device.base.packets.rx": 0,
"kismet.device.base.packets.tx": 0,
"kismet.device.base.packets.llc": 3,
"kismet.device.base.packets.error": 0,
"kismet.device.base.packets.data": 0,
"kismet.device.base.packets.crypt": 0,
"kismet.device.base.packets.filtered": 0,
"kismet.device.base.datasize": 0,
"kismet.device.base.packets.rrd": {
"kismet.common.rrd.last_time": 1559567383,
"kismet.common.rrd.minute_vec": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"kismet.common.rrd.blank_val": 0,
"kismet.common.rrd.aggregator": "default",
"kismet.common.rrd.hour_vec": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"kismet.common.rrd.day_vec": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]
},
"kismet.device.base.signal": {
"kismet.common.signal.type": "dbm",
"kismet.common.signal.last_signal": -56,
"kismet.common.signal.last_noise": 0,
"kismet.common.signal.min_signal": -74,
"kismet.common.signal.min_noise": 0,
"kismet.common.signal.max_signal": -56,
"kismet.common.signal.max_noise": 0,
"kismet.common.signal.maxseenrate": 10,
"kismet.common.signal.encodingset": 1,
"kismet.common.signal.carrierset": 1,
"kismet.common.signal.signal_rrd": {
"kismet.common.rrd.last_time": 1559567383,
"kismet.common.rrd.minute_vec": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"kismet.common.rrd.blank_val": 0,
"kismet.common.rrd.aggregator": "peak_signal"
}
},
"kismet.device.base.freq_khz_map": {
"2437000.000000": 1,
"2442000.000000": 1,
"5500000.000000": 1
},
"kismet.device.base.channel": "6",
"kismet.device.base.frequency": 2442000,
"kismet.device.base.num_alerts": 0,
"kismet.device.base.tags": {
},
"kismet.device.base.seenby": {
"-1970862229": {
"kismet.common.seenby.uuid": "5FE308BD-0000-0000-0000-00C0CAA60413",
"kismet.common.seenby.first_time": 1559567379,
"kismet.common.seenby.last_time": 1559567379,
"kismet.common.seenby.num_packets": 3,
"kismet.common.seenby.freq_khz_map": {
"2437000.000000": 1,
"2442000.000000": 1,
"5500000.000000": 1
},
"kismet.common.seenby.signal": {
"kismet.common.signal.type": "dbm",
"kismet.common.signal.last_signal": -56,
"kismet.common.signal.last_noise": 0,
"kismet.common.signal.min_signal": -74,
"kismet.common.signal.min_noise": 0,
"kismet.common.signal.max_signal": -56,
"kismet.common.signal.max_noise": 0,
"kismet.common.signal.maxseenrate": 10,
"kismet.common.signal.encodingset": 1,
"kismet.common.signal.carrierset": 1,
"kismet.common.signal.signal_rrd": {
"kismet.common.rrd.last_time": 1559567383,
"kismet.common.rrd.minute_vec": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"kismet.common.rrd.blank_val": 0,
"kismet.common.rrd.aggregator": "peak_signal"
}
}
}
},
"kismet.device.base.server_uuid": "A8F71A2C-85F8-11E9-BA41-4B49534D4554",
"dot11.device": {
"dot11.device.typeset": 257,
"dot11.device.client_map": {
},
"dot11.device.num_client_aps": 0,
"dot11.device.advertised_ssid_map": {
"733545801": {
"dot11.advertisedssid.ssid": "LincolnFES-WiFi",
"dot11.advertisedssid.ssidlen": 15,
"dot11.advertisedssid.beacon": 1,
"dot11.advertisedssid.probe_response": 1,
"dot11.advertisedssid.channel": "6",
"dot11.advertisedssid.ht_mode": "HT20",
"dot11.advertisedssid.ht_center_1": 0,
"dot11.advertisedssid.ht_center_2": 0,
"dot11.advertisedssid.first_time": 1559567379,
"dot11.advertisedssid.last_time": 1559567379,
"dot11.advertisedssid.beacon_info": "",
"dot11.advertisedssid.cloaked": 0,
"dot11.advertisedssid.crypt_set": 268436162,
"dot11.advertisedssid.maxrate": 65,
"dot11.advertisedssid.beaconrate": 10,
"dot11.advertisedssid.beacons_sec": 2,
"dot11.advertisedssid.ietag_checksum": 1220416683,
"dot11.advertisedssid.wpa_mfp_required": 0,
"dot11.advertisedssid.wpa_mfp_supported": 0,
"dot11.advertisedssid.dot11d_country": "",
"dot11.advertisedssid.dot11d_list": [
],
"dot11.advertisedssid.wps_state": 0,
"dot11.advertisedssid.dot11r_mobility": 0,
"dot11.advertisedssid.dot11r_mobility_domain_id": 0,
"dot11.advertisedssid.dot11e_qbss": 0,
"dot11.advertisedssid.dot11e_qbss_stations": 0,
"dot11.advertisedssid.dot11e_channel_utilization_perc": 0,
"dot11.advertisedssid.ccx_txpower": 0,
"dot11.advertisedssid.cisco_client_mfp": 0,
"dot11.advertisedssid.ie_tag_list": [
0,
1,
3,
5,
42,
50,
48,
45,
61,
127,
221
]
}
},
"dot11.device.num_advertised_ssids": 1,
"dot11.device.probed_ssid_map": {
},
"dot11.device.num_probed_ssids": 0,
"dot11.device.associated_client_map": {
},
"dot11.device.num_associated_clients": 0,
"dot11.device.client_disconnects": 0,
"dot11.device.last_sequence": 0,
"dot11.device.bss_timestamp": 0,
"dot11.device.num_fragments": 0,
"dot11.device.num_retries": 0,
"dot11.device.datasize": 0,
"dot11.device.datasize_retry": 0,
"dot11.device.last_probed_ssid_csum": 0,
"dot11.device.last_beaconed_ssid": "LincolnFES-WiFi",
"dot11.device.last_beaconed_ssid_checksum": 733545801,
"dot11.device.last_bssid": "38:D2:69:F5:B4:AF",
"dot11.device.last_beacon_timestamp": 1559567379,
"dot11.device.wps_m3_count": 0,
"dot11.device.wps_m3_last": 0,
"dot11.device.wpa_handshake_list": [
],
"dot11.device.wpa_nonce_list": [
],
"dot11.device.wpa_anonce_list": [
],
"dot11.device.wpa_present_handshake": 0,
"dot11.device.min_tx_power": 0,
"dot11.device.max_tx_power": 0,
"dot11.device.supported_channels": [
],
"dot11.device.link_measurement_capable": 0,
"dot11.device.neighbor_report_capable": 0,
"dot11.device.extended_capabilities": [
],
"dot11.device.beacon_fingerprint": 4212996422,
"dot11.device.probe_fingerprint": 0,
"dot11.device.response_fingerprint": 0
}
}
When you want to recursively walk through the fields of an entire object and its contents, you need json_tree():
SELECT j.value
FROM devices AS d
JOIN json_tree(d.device) AS j
WHERE j.key = 'dot11.advertisedssid.ssid';
gives
value
--------------
SampleFES-WiFi
when run on a table holding a fixed version of that sample object.
I know this is a bit old, but OP seemed (in comments) to want a more complete solution. I know I did when I first came across this answer. The accepted solution allows you to pull in one field from the JSON blob, but the common use case in OP's example is to pull multiple fields from that blob. After some searching I found that the json_extract() function works very well for this once you realize that the "dot11.device.advertised_ssid_map" object is an array. Once you provide it with an index his normal query method works.
Considerations:
OP's example is relating to the Kismet device field in the devices table, so my example will use a common query that I often need in the context of that table
With Kismet the keys used in these JSON blobs are long and contain dots, so the syntax for specifying them in SQLite3 is a bit cumbersome for some nested values
SQLite3's JSON1 extension does not seem to like some of the wildcarding syntax normally allowed in JSONPath specifications, so long explicit paths are required
So here is my solution:
SELECT devmac, strongest_signal,
json_extract(d.device, '$."dot11.device"."dot11.device.advertised_ssid_map"[0]."dot11.advertisedssid.ssid"') AS ssid,
json_extract(d.device, '$."dot11.device"."dot11.device.advertised_ssid_map"[0]."dot11.advertisedssid.cloaked"') AS cloaked,
json_extract(d.device, '$."kismet.device.base.signal"."kismet.common.signal.min_signal"') AS weakest_signal,
json_extract(d.device, '$."kismet.device.base.channel"') AS channel,
json_extract(d.device, '$."dot11.device"."dot11.device.num_associated_clients"') AS clientCnt,
json_extract(d.device, '$."kismet.device.base.crypt"') AS crypt,
json_extract(d.device, '$."kismet.device.base.manuf"') AS manuf
FROM devices AS d
WHERE type = 'Wi-Fi AP'
;
I am trying to figure out how to do this transformation symbolically in theano a matrix of undetermined size
From:
[[0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1],
.
.
]
To:
[[1, 2, 3, 0, 1, 2, 3, 4, 5, 0, 0, 1, 0, 1, 2, 3, 0],
[1, 2, 3, 4, 5, 6, 7, 8, 0, 1, 2, 0, 0, 0, 0, 0, 0],
.
.
]
So for every consecutive 0 I want an increasing range and whenever I stumble on a 1 the range resets.
Here's one way to do it, using inefficient scans:
import theano
import theano.tensor as tt
def inner_step(x_t_t, y_t_tm1):
return tt.switch(x_t_t, 0, y_t_tm1 + 1)
def outer_step(x_t):
return theano.scan(inner_step, sequences=[x_t], outputs_info=[0])[0]
def compile():
x = tt.bmatrix()
y = theano.scan(outer_step, sequences=[x])[0]
return theano.function([x], y)
def main():
f = compile()
data = [[0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1]]
print f(data)
main()
When run, this prints:
[[1 2 3 0 1 2 3 4 5 0 0 1 0 1 2 3 0]
[1 2 3 4 5 6 7 8 0 1 2 0 0 0 0 0 0]]
I finally finished building my Wavefront OBJ parser, but still have some issues rendering my test object (cube).
So this is how i have parsed my vertices and indices (faces) into arrays of data. I have ignored textures and normals for now.
Vertices:
v -0.307796 0.00433517 0
v 0.299126 0.00433517 0
v 0.299126 0.00433517 0.48337
v -0.307796 0.00433517 0.48337
v -0.307796 0.364153 0.48337
v 0.299126 0.364153 0.48337
v 0.299126 0.364153 0
v -0.307796 0.364153 0
As:
const Vertex Vertices[] = {
{-0.307796,0.00433517,0},
{0.299126,0.00433517,0},
{0.299126,0.00433517,0.48337},
{-0.307796,0.00433517,0.48337},
{-0.307796,0.364153,0.48337},
{0.299126,0.364153,0.48337},
{0.299126,0.364153,0},
{-0.307796,0.364153,0}
};
Faces:
f 7/1/1 3/2/2 2/3/3
f 3/4/4 7/5/5 6/6/6
f 5/7/7 1/8/8 4/9/9
f 1/10/10 5/11/11 8/12/12
f 7/13/13 1/14/14 8/15/15
f 1/16/16 7/17/17 2/18/18
f 3/19/19 5/20/20 4/21/21
f 5/22/22 3/23/23 6/24/24
f 5/25/25 7/26/26 8/27/27
f 7/28/28 5/29/29 6/30/30
f 3/31/31 1/32/32 2/33/33
f 1/34/34 3/35/35 4/36/36
As:
const GLubyte Indices[] = {
7,1,1, 3,2,2, 2,3,3,
3,4,4, 7,5,5, 6,6,6,
5,7,7, 1,8,8, 4,9,9,
1,10,10, 5,11,11, 8,12,12,
7,13,13, 1,14,14, 8,15,15,
1,16,16, 7,17,17, 2,18,18,
3,19,19, 5,20,20, 4,21,21,
5,22,22, 3,23,23, 6,24,24,
5,25,25, 7,26,26, 8,27,27,
7,28,28, 5,29,29, 6,30,30,
3,31,31, 1,32,32, 2,33,33,
1,34,34, 3,35,35, 4,36,36
};
Indices only as vertex positions:
const GLubyte Indices[] = {
7, 3, 2,
3, 7, 6,
5, 1, 4,
1, 5, 8,
7, 1, 8,
1, 7, 2,
3, 5, 4,
5, 3, 6,
5, 7, 8,
7, 5, 6,
3, 1, 2,
1, 3, 4
};
SetupVBO:
GLuint vertexBuffer;
glGenBuffers(1, &vertexBuffer);
glBindBuffer(GL_ARRAY_BUFFER, vertexBuffer);
glBufferData(GL_ARRAY_BUFFER, sizeof(Vertices), Vertices, GL_STATIC_DRAW);
GLuint indexBuffer;
glGenBuffers(1, &indexBuffer);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);
glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(Indices), Indices, GL_STATIC_DRAW);
Renderingcode:
glClearColor(0, 104.0/255.0, 55.0/255.0, 1.0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glEnable(GL_DEPTH_TEST);
CC3GLMatrix *projection = [CC3GLMatrix matrix];
float h = 4.0f * self.frame.size.height / self.frame.size.width;
[projection populateFromFrustumLeft:-2 andRight:2 andBottom:-h/2 andTop:h/2 andNear:4 andFar:10];
glUniformMatrix4fv(_projectionUniform, 1, 0, projection.glMatrix);
CC3GLMatrix *modelView = [CC3GLMatrix matrix];
[modelView populateFromTranslation:CC3VectorMake(sin(CACurrentMediaTime()), 0, -7)];
_currentRotation += displayLink.duration * 90;
[modelView rotateBy:CC3VectorMake(_currentRotation, _currentRotation, 0)];
glUniformMatrix4fv(_modelViewUniform, 1, 0, modelView.glMatrix);
// 1
glViewport(0, 0, self.frame.size.width, self.frame.size.height);
// 2
glVertexAttribPointer(_positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
glVertexAttribPointer(_colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (GLvoid*) (sizeof(float) * 3));
// 3
glDrawElements(GL_TRIANGLES, sizeof(Indices)/sizeof(Indices[0]), GL_UNSIGNED_BYTE, 0);
And the result is pretty much nothing at all, so there must be something that im doing completely wrong.
Furthermore will the indices cause any problems since i have not initialized the normals or texture coords?
The Wavefront obj format says that if you have sequences of 3 in the face definitions then their meaning is:
vertex-index/vertex-texture-index/vector-normal-index
You are reading all of the indices into a single array GLubyte Indices[] and using it as it if was just the indices of the vertices.
If you want to do away with textures and normals, you need to take only the first number of every triplet.
In
f 6/1/1 3/2/2 2/3/3 7/4/4
The face is a quad of vertices of indices [6,3,2,7]. Using the indices array like you have requires that you tell OpenGl that the indices are multiplexed in triplets. It does not look like you do that. It also requires additional buffers for normal and texture coordinates.