Have you ever taken a close look at some of the urls produced as you browse around www.amazon.com? Here's a typical one:
http://www.amazon.com/s/ref=sr_nr_p_n_age_range_2?rh=i%3Aaps%2Ck%3Abumblebee%2Cp_n_age_range%3A165890011%7C165936011&keywords=bumblebee&ie=UTF8&qid=1373589657&rnid=165794011
After you urldecode it, you get:
http://www.amazon.com/s/ref=sr_nr_p_89_2?rh=i:aps,k:bumblebee,p_n_age_range:165890011|165936011,p_89:Pillow Pets&keywords=bumblebee&ie=UTF8&qid=1373589678&rnid=2528832011
I'm particularly interested in this portion:
rh=i:aps,k:bumblebee,p_n_age_range:165890011|165936011,p_89:Pillow Pets
It looks like some kind of object serialization that reduces down even smaller than JSON. There are no quotes around strings, and arrays are bracketless and delimited by pipes.
Ebay does something similar, particularly the array pipes. Have you ever seen this format? Is this a common pattern you recognize? Or is it just some exotic hand-rolled solution that they both happened to arrive at independently? Any idea why they do it, as opposed to just straight JSON?
Related
A little while ago, I started learning Kotlin, and I have done its basics, variables, classes, lists, and arrays, etc. but the book I was learning from seemed to miss one important aspect, reading and writing to a file, maybe a function like "fwrite" in C++
So I searched google, and yes, reading and writing bytes were easy enough. However, I being used to C++'s open personality, wanted to make a "kind of" database.
In C++ I would simply make a struct and keep appending it to a file, and then read all the stored objects one by one, by placing "fread" in a for loop or just reading into an array of the struct in one go, as the struct was simply just the bytes allocated to the variables inside it.
However in Kotlin, there is no struct, instead, we use Data Class to group data. I was hoping there was an equally easy way to store data in a file in form of Data Class and read it into maybe a List of that class, or if that is not possible, maybe some other way to store grouped data that would be easy to read and write.
Easiest way is to use a serialization library. Kotlin already provides something for that
TL;DR;
Add KotlinX Serialization to your project, choose the serialization format you prefer (protobuf or cbor will fit, go for json if you prefer something more human readable although bigger in size), use the proper serializer for generating your ByteArray and write it to a file using Kotlin methods for that
Generating the ByteArray might be tricky, not sure as I'm telling this from memory. What I can tell for sure is that if you choose JSON you can get the string representation and write to a file. So I'm assuming the same will be valid for binary formats (but writing to a file in binary instead of strings)
What you need can be fulfilled by ROOM DATABASE
It is officially recommended by GOOGLE, It uses your Android application's internal Database which is made using SQLITE
You can read more info about ROOM at
https://developer.android.com/jetpack/androidx/releases/room?gclid=Cj0KCQjw5ZSWBhCVARIsALERCvwjmJqiRPAnSYjhOzhPXg8dJEYnqKVgcRxSmHRKoyCpnWAPQIWD4gAaAlBnEALw_wcB&gclsrc=aw.ds
It provided Data Object Class (DAO) and Entity Classes through which one can access the database TABLE using SQL Queries.
Also, it will check your queries at compile time for any errors in it.
Note: You need to have basic SQL Knowledge for building the queries for CRUD Operations
I'm very new to Go and am currently porting a PHP program.
I understand that Go is not a dynamically-typed language and I like that about it. It seems very structured and easy to keep track of everything.
But I've been coming across situations that seem to be a little ... ugly. Is there a better way of performing this sort of process:
plyr := builder.matchDetails.plyr[i]
plyrDetails := strings.Split(plyr, ",")
details := map[string]interface{}{
"position": plyrDetails[0], "id": plyrDetails[1],
"xStart": plyrDetails[2], "zStart": plyrDetails[3],
}
EDIT:
Is there a better way to achieve a map containing the strings from plyr than to create two additional variables, to be destroyed straight afterwards? Or is this the correct way?
tl;dr:
If possible, choose a different format and let a library do the string parsing/generation for you
Use structs rather than maps for anything you use a few times, for more compiler checks
The common way of using encoding/json accomplishes both of those.
Meanwhile, don't sweat perf too much because you'll probably vastly improve the old app's speed regardless; there's no indication speed of parsing or GC is a problem yet; and the syntactical differences you mentioned in the first rev. of the post don't necessarily actually relate to GC.
So, I understand you may be porting piece-for-piece, and that may limit what you can change now.
But if/when you can change things, a really clean solution would be to use the encoding/json package and a struct: the json package will parse input/generate output from structs without any manual string manipulation on your part, and using a struct gives you compile-time checking rather than only the runtime checking you get with a map. Lots of Go apps (and others) use JSON to expose their services.
An intermediate step could be to introduce struct types for any internal structure you use at least a few times, rather than maps, so even without updating the parsing, at least the internals of the app get the benefits of compile-time checking. structs are also what things like the gorm object/relational mapper expect to deal with. They happen to use less memory than maps, and be quicker (and more concise syntactically) to access, but those aren't even necessarily the most important considerations here.
On the performance of what you have now, and particularly whether different syntax would make it faster: don't sweat that, for a bunch of reasons: the port's likely to be faster than the PHP was whatever you do; we don't yet have any indication that parsing or GC is actually slow or your bottleneck; and the syntactical differences you talked about in the first revision of your question may not relate to GC much or at all. More/fewer var names in your code may not correspond to more/fewer heap allocations, 'cause often Go can allocate on the stack, briefly discussed under 'escape analysis' in Dave Cheney's Gocon Tokyo slides. And as peterSO said, we seem to be looking at allocations of smallish references, not, say, copying all of the string bytes from the request each time.
Go is NOT PHP. Write Go programs in Go. Write PHP programs in PHP.
Interface values are represented as a two-word pair giving a pointer
to information about the type stored in the interface and a pointer to
the associated data. Go Data Structures:
Interfaces
Reusing Go interface variables to "increase performance" makes no sense.
I'm working on a first JSON-RPC/JSON-REST API. One of the conveniences of JSON is that it can easily represent structured data (a user may have multiple email addresses, multiple addresses), etc...
For example, the Facebook Graph API nicely represents the kind of thing that's handy to return as JSON objects:
https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-ash3/851559_339008529558010_1864655268_n.png
However, in implementing an API such as this with a relational database, we end up shattering structured objects into very many tables (at least one for each list in the JSON object), and un-shattering them when responding to requests. So:
requires a lot of modelling (separate models for JSON object and SQL tables).
inconsistencies creep in between the models: e.g. user_id (in SQL) vs. userID (in JSON)
marshaling stuff between one model and the other is very time consuming (tedious, error-prone and pointless boilerplate).
What design-patterns exist to help in this situation?
I'm not sure you are looking for design patterns. I would look for tools that handle this better.
I assume that you want to be able to query these objects, and not just store them in TEXT fields. Many databases support XML fairly well, so I would convert the JSON to XML (with a library) and then store that in the database.
You may also want to consider a JSON document based database. That will definitely get you where you want to go.
If you don't need to be able to query these, or only need to query a very small subset of fields, just store the objects as text, and extract those query-able fields into actual columns. This way you don't need to touch the majority of the data, but you can still query the few fields you care about. (Plus you can index them for speedier lookup.)
I have always chosen to implement this functionality in a facade pattern. Since the point of the facade is to simplify (abstract) an underlying complexity as a boundary between two or more systems, it seemed like the perfect place to handle this.
I realize however that this does not quite answer the question. I am talking about the container for the marshalling while the question is about how to better manage the contents (the code that does the job).
My approach here is somewhat old fashioned, but since this an old question maybe that’s okay. I employ (as much as possible) stored procedures in the dB. This promotes better encapsulation than one typically finds with a code layer outside of the dB that has to “know about” dB structure. What inevitably happens in the latter case is that more than one system will be written to do this (one large company I worked at had at least 6 competing ESBs) and there will be conflicts. Also, usually the stored procedure scripting will benefit from some sort of IDE that will helps maintain contextual awareness of the dB structure.
So this approach - even though it is not a pattern per se - makes managing the ORM a lot easier.
I currently have an API endpoint which returns an array of objects each containing 4 variables in JSON format.
The size of the data ranges from 500kb to 5mb - depending on the number of records. In order to reduce the size of the return, we're considering removing the labels from the objects, and returning an array of arrays.
E.g.
[{propertyone:123,propertytwo:456,propertythree:789,propertyfour:012},etc,etc,etc]
would become
[[123,456,789,012],etc,etc,etc]
We would then document that array position 0 is to considered propertyone, etc. There may be a point in the future when this API becomes public. Is it considered better practice to leave the names in, or would serving a documented API with an enforced order suffice.
Well documented APIs are often the best APIs. Having external documentation for your API is important, but the data you return needs to be documented as well, and nothing beats self-documenting data.
I think by removing the property names, you're going to lose this self-documenting entirely and you could end up limiting yourself in the future as well. For example, consider only wanting to return partial data, how would you represent it without those property names?
In addition to the documentation, if you're returning JSON then depending on the consumer's environment, providing those properties could give them a natural model to work with. Consider JavaScript as an example, if they simply parse your response into JavaScript objects they essentially have a client-side model implemented for free and that will make it a joy to work with your API.
Nonetheless, speed is important and I would suggest looking into different measures for reducing the size of the return. Often a common solution to this is providing paged results, requiring a client to perform additional requests if they desire more data. Also, depending on the shape of your data, you could benefit greatly by using something as simple as GZIP compression on your responses.
While designing an API, as long as you document it and it works as described, normally anything will suffice. However, the API that empowers the consumer is the one that succeeds.
I'm creating a vb.net winforms application that will take in user given strings, parse them, and print out labels with variable information. The given string will be used in all the labels, but the variable part of the string will change with each label.
My question is: is it better to parse the strings one time, then store those values in arrays, or to parse the string each time a label is printed? Which will perform better? Which is better practice? What is the proper way to test something like this?
If saving data in memory is done for performance, my preference would be not to do it until I knew the parsing was actually a performance problem, which I detect by random-pausing.
Generally my rule of thumb is - the less data structure the better.
If you have data structure you have to worry about it getting stale, and any time you change the program you have more code to modify and put bugs into.
Besides, parsing does not need to be slow, especially compared to whatever else you're doing.
Parsing is definitely heavy stuff.
In this particular instance if you do not have any memory constraints then you should parse once and store to speed up your application, especially if it is going to be something that is being used by people :)