I've had trouble printing the values of the key "uuid". The "uuid" key shows up multiple times throughout the file whilst other keys that are only present in the file once have no trouble being printed. So I'm wondering is it possible to do what I want it to do? The error I'm getting is a KeyError: 'uuid' for your information.
path = os.path.join(startn, listn + endn)
with open(path, encoding='utf-8') as json_file:
auction = json.load(json_file)
print("Type:", type(auction))
print("\nAuction:", auction['uuid'])
Also the file data looks like this
{"uuid":"36ff18f6e56d49b18c55cd06df3dfce8","auctioneer":"7c1251d409524cfd96b68da183698676","profile_id":"b7c111408b7c4d57a0665edda28c3b77"}
{"uuid":"754c3f2a25d949d1907f9c29f761b636","auctioneer":"f281bf681baa4cfea8a798cbe76c15f3","profile_id":"f281bf681baa4cfea8a798cbe76c15f3"}
etc...
Given that you haven't given us the full json but rather a messed up one
{"uuid":"36ff18f6e56d49b18c55cd06df3dfce8","auctioneer":"7c1251d409524cfd96b68da183698676","profile_id":"b7c111408b7c4d57a0665edda28c3b77"} {"uuid":"754c3f2a25d949d1907f9c29f761b636","auctioneer":"f281bf681baa4cfea8a798cbe76c15f3","profile_id":"f281bf681baa4cfea8a798cbe76c15f3"}
I assume that the actual json is something like this
[
{
"uuid": "36ff18f6e56d49b18c55cd06df3dfce8",
"auctioneer": "7c1251d409524cfd96b68da183698676",
"profile_id": "b7c111408b7c4d57a0665edda28c3b77"
},
{
"uuid": "754c3f2a25d949d1907f9c29f761b636",
"auctioneer": "f281bf681baa4cfea8a798cbe76c15f3",
"profile_id": "f281bf681baa4cfea8a798cbe76c15f3"
}
]
This is an array of json. What python is returning is an array, which is not accessed by key. You need to iterate the array like so
with open(path, encoding='utf-8') as json_file:
// Changed the return as plural because this is an array
auctions = json.load(json_file)
// Iterate through your array
for auction in auctions:
print("Type:", type(auction))
print("\nAuction:", auction['uuid'])
Related
I have started using Pymongo recently and now I want to find the best way to remove $oid in Response
When I use find:
result = db.nodes.find_one({ "name": "Archer" }
And get the response:
json.loads(dumps(result))
The result would be:
{
"_id": {
"$oid": "5e7511c45cb29ef48b8cfcff"
},
"about": "A jazz pianist falls for an aspiring actress in Los Angeles."
}
My expected:
{
"_id": "5e7511c45cb29ef48b8cfcff",
"about": "A jazz pianist falls for an aspiring actress in Los Angeles."
}
As you seen, we can use:
resp = json.loads(dumps(result))
resp['id'] = resp['id']['$oid']
But I think this is not the best way. Hope you guys have better solution.
You can take advantage of aggregation:
result = db.nodes.aggregate([{'$match': {"name": "Archer"}}
{'$addFields': {"Id": '$_id.oid'}},
{'$project': {'_id': 0}}])
data = json.dumps(list(result))
Here, with $addFields I add a new field Id in which I introduce the value of oid. Then I make a projection where I eliminate the _id field of the result. After, as I get a cursor, I turn it into a list.
It may not work as you hope but the general idea is there.
First of all, there's no $oid in the response. What you are seeing is the python driver represent the _id field as an ObjectId instance, and then the dumps() method represent the the ObjectId field as a string format. the $oid bit is just to let you know the field is an ObjectId should you need to use for some purpose later.
The next part of the answer depends on what exactly you are trying to achieve. Almost certainly you can acheive it using the result object without converting it to JSON.
If you just want to get rid of it altogether, you can do :
result = db.nodes.find_one({ "name": "Archer" }, {'_id': 0})
print(result)
which gives:
{"name": "Archer"}
import re
def remove_oid(string):
while True:
pattern = re.compile('{\s*"\$oid":\s*(\"[a-z0-9]{1,}\")\s*}')
match = re.search(pattern, string)
if match:
string = string.replace(match.group(0), match.group(1))
else:
return string
string = json_dumps(mongo_query_result)
string = remove_oid(string)
I am using some form of custom handler. I managed to remove $oid and replace it with just the id string:
# Custom Handler
def my_handler(x):
if isinstance(x, datetime.datetime):
return x.isoformat()
elif isinstance(x, bson.objectid.ObjectId):
return str(x)
else:
raise TypeError(x)
# parsing
def parse_json(data):
return json.loads(json.dumps(data, default=my_handler))
result = db.nodes.aggregate([{'$match': {"name": "Archer"}}
{'$addFields': {"_id": '$_id'}},
{'$project': {'_id': 0}}])
data = parse_json(result)
In the second argument of find_one, you can define which fields to exclude, in the following way:
site_information = mongo.db.sites.find_one({'username': username}, {'_id': False})
This statement will exclude the '_id' field from being selected from the returned documents.
I've got an API endpoint POST /data.
The received data is formatted in a certain way which is different from the way I store it in the db.
I'll use geometry type from postgis as an example.
class MyPostgisModel(models.Model):
...
position = models.PointField(null=True)
my_charfield = models.CharField(max_length=10)
...
errors = JSONField() # Used to save the cleaning and validation errors
class MyPostgisSerializer(serializers.ModelSerializer):
class Meta:
model = MyPostgisModel
fields = [
...
"position",
...
"my_charfield",
"errors",
]
def to_internal_value(self, data):
...
# Here the data is coming in the field geometry but in the db, it's called
# position. Moreover I need to apply the `GEOSGeometry(json.dumps(...))`
# method as well.
data["position"] = GEOSGeometry(json.dumps(data["geometry"]))
return data
The problem is that there is not only one field like position but many. And I would like (maybe wrongly) to do like the validate_*field_name* scheme but for cleaning (clean_*field_name*).
There is another problem. In this scheme, I would like to still save the rest of the data in the database even if some fields have raised ValidationError (eg: a CharField that is too long) but are not part of the primary_key/a unique_together constraint. And save the related errors into a JSONField like this:
{
"cleaning_errors": {
...
"position": 'Invalid format: {
"type": "NotAValidType", # Should be "Point"
"coordinates": [
4.22,
50.67
]
}'
...
},
"validating_errors": {
...
"my_charfield": "data was too long: 'this data is way too long for 10 characters'",
...
}
}
For the first problem, I thought of doing something like this:
class BaseSerializerCleanerMixin:
"""Abstract Mixin that clean fields."""
def __init__(self, *args, **kwargs):
"""Initialize the cleaner strategy."""
# This is the error_dict to be filled by the `clean_*field_name*`
self.cleaning_error_dict = {}
super().__init__(*args, **kwargs)
def clean_fields(self, data):
"""Clean the fields listed in self.fields_to_clean before validating them."""
cleaned_data = {}
for field_name in getattr(self.Meta, "fields", []):
cleaned_field = (
getattr(self, "clean_" + field_name)(data)
if hasattr(self, "clean_" + field_name)
else data.get(field_name)
)
if cleaned_field is not None:
cleaned_data[field_name] = cleaned_field
return cleaned_data
def to_internal_value(self, data):
"""Reformat data to put it in the database."""
cleaned_data = self.clean_fields(data)
return super().to_internal_value(cleaned_data)
I'm not sure that's a good idea and maybe there is an easy way to deal with such things.
For the second problem ; catching the errors of the validation without specifying with is_valid() returning True when no primary_key being wrongly formatted, I'm not sure how to proceed.
I'm trying to remove an object from an array of objects if a certain value exists in nested data.
The data being returned from the API is shaped like this:
Array [
Object {
"id": "/db/Shifts/123",
"applicants": Object {
"applicants": Array [
"/db/User/12",
"/db/User/13",
],
},
Object {
"id": "/db/Shifts/456",
"applicants": Object {
"applicants": Array [
"/db/User/12",
"/db/User/14",
],
},
Object {
"id": "/db/Shifts/789",
"applicants": Object {
"applicants": Array [
"/db/User/13",
"/db/User/14",
],
},
]
Using Ramda, how would I filter out the shifts where User 12 exists in the array of applicants, which would be located at applicants.applicants.
I am not able to flatten the data in this case, the list of applicants for each shift does have to be an array contained in an object.
I tried this:
var hasApplied = pathEq(['applicants', 'applicants'], 'db/User/12');
console.log(filter(hasApplied, shifts));
But I don't think that is quite right because applicants.applicants is an array, I feed like I need to be feeding it one more function to get into the array of applicants but I'm not sure what.
Your use of R.pathEq is resulting in the user ID being compared with each array of IDs for equality, rather than checking whether each array contains the given ID.
You could instead make use of R.pathSatisfies along with R.contains
const hasApplied = R.pathSatisfies(R.contains('/db/User/12'), ['applicants', 'applicants'])
I'm trying to use angular-datatables with serverside processing. However, it seems that angular-datatables expects that the data from the server is in object format (object vs array data described) with column names preceding each table datapoint. I'd like to configure angular-datatables to accept array based data since I can't modify my server side output which only outputs data in array format.
I'm configuring Datatables in my javascript like so:
var vm = this;
vm.dtOptions = DTOptionsBuilder.newOptions()
.withOption('ajax', {
url: 'output/ss_results/' + $routeParams.uuid,
type: 'GET'
})
.withDataProp('data')
.withOption('processing', true)
.withOption('serverSide', true);
My data from the server looks like this in array format:
var data = [
[
"Tiger Nixon",
"System Architect",
"$3,120"
],
[
"Garrett Winters",
"Director",
"$5,300"
]
]
But as far as I can tell, angular-datatables is expecting the data in object format like so:
[
{
"name": "Tiger Nixon",
"position": "System Architect",
"extn": "5421"
},
{
"name": "Garrett Winters",
"position": "Director",
"extn": "8422"
}
]
I tried not defining dtColumns or setting it to an empty array like vm.dtColumns = []; but I get an error message when I do that. When I configure dtColumns with a promise to load the column data via ajax I get datatables error #4 because it can't find the column name preceding my table datapoints in the data retrieved from the server.
Is it possible to configure angular-datatables to accept array based data? I can't find anything on the angular-datatables website that indicates it can be configured this way.
Edit: So I removed the .withDataProp('data') which I think was causing the problem. The table works a little better now but it's still broken. After it loads, I get the message No matching records found. Even though right below it it says Showing 1 to 10 of 60,349 entries
Previous1…456…6035Next Does anyone know why this might be?
If you want to use an array of arrays instead of an array of objects, simply refer to the array indexes instead of the object names :
$scope.dtColumns = [
DTColumnBuilder.newColumn(0).withTitle('Name'),
DTColumnBuilder.newColumn(1).withTitle('Position'),
DTColumnBuilder.newColumn(2).withTitle('Office'),
DTColumnBuilder.newColumn(3).withTitle('Start date'),
DTColumnBuilder.newColumn(4).withTitle('Salary')
]
demo using the famous "Tiger Nixon" array loaded via AJAX -> http://plnkr.co/edit/16UoRqF5hvg2YpvAP8J3?p=preview
Question
How do I (with a single HTTP request to the REST API) write an array to Firebase and give each array element a (non-integer) unique ID?
As described here.
Data
The data I have to write looks like the following.
data-to-write.js
myArray = [ {"user_id": "jack", "text": "Ahoy!"},
{"user_id": "jill", "text": "Ohai!"} ];
Goal
When finished, I want my Firebase to look like this following.
my-firebase.firebaseio.com
{
"posts": {
"-JRHTHaIs-jNPLXOQivY": { // <- unique ID (non-integer)
"user_id": "jack",
"text": "Ahoy!"
},
"-JRHTHaKuITFIhnj02kE": { // <- unique ID (non-integer)
"user_id": "jill",
"text": "Ohai!"
}
}
}
I do not want it to look like this following...
my-anti-firebase.firebaseio.com
// NOT RECOMMENDED - use push() instead!
{
"posts": {
"0": { // <- ordered array index (integer)
"user_id": "jack",
"text": "Ahoy!"
},
"1": { // <- ordered array index (integer)
"user_id": "jill",
"text": "Ohai!"
}
}
}
I note this page where it says:
[...] if all of the keys are integers, and more than half of the keys between 0 and the maximum key in the object have non-empty values, then Firebase will render it as an array.
Code
Because I want to do this in a single HTTP request, I want to avoid iterating over each element in the array and, instead, I want to push a batch in a single request.
In other words, I want to do something like this:
pseudocode.js
curl -X POST -d '[{"user_id": "jack", "text": "Ahoy!"},
{"user_id": "jill", "text": "Ohai!"}]' \
// I want some type of batch operation here
'https://my-firebase.firebaseio.com/posts.json'
However, when I do this, I get exactly what I describe above that I don't want (i.e., sequential integer keys).
I want to avoid doing something like this:
anti-pseudocode.js
for(i=0; i<=myArray.length; i++;){ // I want to avoid iterating over myArray
curl -X POST -d '{"user_id": myArray[i]["user_id"],
"text": myArray[i]["text"]}' \
'https://my-firebase.firebaseio.com/posts.json'
}
Is it possible to accomplish what I have described? If so, how?
I don't think there is a way to use the Firebase API to do this as described in the OP.
However, it can be done with a server script as follows:
Iterate through each array element.
Assign each element a unique id (generated by server script).
Create a return object with keys being the unique IDs and values being the corresponding array elements.
Write object to Firebase with a single HTTP request using the patch method. Because post creates a new Firebase generated ID for the entire object itself. Whereas, patch does not; it writes directly to the parent node.
script.js
var myObject = {},
i = myArray.length;
while(i--){
var key = function(){ /* return unique ID */ }();
myObject[key] = myArray[i];
}
curl -X PATCH -d JSON.stringify(myObject) \
'https://my-firebase.firebaseio.com/posts.json'
Your decision to use POST is correct. The one which cause numeric indexes as a key is because your payload is an array. Whenever you post/put and array, the key will always be indexes. Post your object one by one if you want the server generate key for you.
Firebase will generate unique ID only if you use POST. If you use PATCH unique ID is not generated.
Hence for the given case, will need to iterate through using some server/ client side code to save data in firebase.
Peseudo Code:
For each array
curl -X POST -d
"user_id": "jack",
"text": "Ahoy!"
'https://my-firebase.firebaseio.com/posts.json'
Next