I have come across various situations, where I want to store some formatted data in a way such that it can be easily queried.
For example
$ cat so.txt
"question_id": 58640,
"tags": ["polls", "fun", "quotes"],
"title": "Great programming quotes"
"question_id": 184618,
"tags": ["polls", "fun", "comment"],
"title": "What is the best comment in source code you have ever encountered?"
"question_id": 3734102,
"tags": ["c++", "linux", "exit-code"],
"title": "Why cant' I return bigger values from main function ?"
"question_id": 2349378,
"tags": ["communication", "terminology", "vocabulary"],
"title": "New programming jargon you coined?"
"question_id": 3723817,
"tags": ["open-source", "project-management", "failure", "fail"],
"title": "How to make an open source project fail"
"question_id": 3699150,
"tags": ["testing", "interview-questions", "job-interview"],
"title": "Interview question please help"
$
A simple query can be displaying the titles of the questions with tags "C++".
These are the requirements
The database has to support just me.
It must be able to support all the general SQL type queries. I am familiar with SQL, so the more it is SQL-like, the better.
It has to be run locally on my Linux machine running Ubuntu 10.04.
Also, since my requirements are minimal, I expect it not to use up too much memory.
What DBMS do you suggest for this purpose?
the code you posted looks like json. if this is the format you primarily keep your data, it's possible that something like couchdb would be ideal for you? I'm a big fan of it, so I'm biased :)
otherwise, the traditional answer to "light database that only supports me" is SQLite.
Related
Context
In our project, we need to represent resources defined by the users. That is, every user can have different resources, with different fields, different validations, etc. So we have two different things to represent in our API:
Resource definition: this is just a really similar thing to a json schema, it contains the fields definitions of the resource and its limitations (like min and max value for numeric fields). For instance, this could be the resource definition for a Person:
{
"$id": "https://example.com/person.schema.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Person",
"type": "object",
"properties": {
"firstName": {
"type": "string",
"description": "The person's first name."
},
"lastName": {
"type": "string",
"description": "The person's last name."
},
"age": {
"description": "Age in years which must be equal to or greater than zero.",
"type": "integer",
"minimum": 0
}
}
}
Resource instance: this is just an instance of the specified resource. For instance, for the Person resource definition, we can have the following instances:
[
{
"firstName": "Elena",
"lastName": "Gomez",
},
{
"firstName": "Elena2",
"lastName": "Gomez2",
},
]
First opinion
So, it seems this kind of presents some conflicts with the Restful API approach. In particular, I think it has some problems with the Uniform Interface. When you get a resource, you should be able to handle the resource without any additional information. With this design, you need to make an additional request to first get the resource definition. Let's see this with an example:
Suppose you are our web client. And you are logged in as an user with the Person resource. To show a person in the UI, you first need to know the structure of the Person resource, that is, you to do the following request: GET /resource_definitions/person. And then, you need to request the person object: GET /resource/person/123.
Second opinion
Others seem to think that this is not a problem and that the design is still RESTful. Every time you ask for something to an API, you need to know the format previously, is not self-documented in the API, so it makes sense for this endpoint to behave the same as the others.
Question
So what do you think? Is the proposed solution compliance with the RESTful approach to API design?
The simple solution is to add a link:
{
"_links": {
"describedby": {
"href": "https://example.com/person.schema.json",
"type": "application/schema+json"
}
},
"firstName": "Elena",
"lastName": "Gomez"
}
You could also put this in a header. This is semantically equivalent:
Link: <https://example.com/person.schema.json>; rel="describedby" type="application/schema+json"
It does not violate the uniform interface if there is no standard for this kind of stuff, but there is. RDF e.g. JSON-LD and schema.org vocab can handle most of these types. Even for REST there is an RDF vocab called Hydra, though the community is not that active nowadays.
As of the actual problem, I would look around, maybe RDF technologies or graph technologies are better for it, though I am not sure how much connection there is in your graph. If it is just a few types and instances, then I would probably stick to REST.
Ohh I see meanwhile, you used an actual JSON schema. Then that part is certainly uniform interface compatible. As of the instances you need to add something like type: "https://example.com/person.schema.json" and you are ok. Maybe a vendor specific JSON derived MIME type which describes what "type" means in this context if you want to be super precise or just use JSON-LD instead. https://www.w3.org/2019/wot/json-schema Or an alternative more common solution is using RDFS and XSD with JSON-LD instead of JSON Schema.
I'm trying to structured data for producing the review like this on google search (please see the image) -
According to this link I've to write the following structured data markup -
<script type="application/ld+json">
{
"#context": "http://schema.org/",
"#type": "Review",
"itemReviewed": {
"#type": "Thing",
"name": "Super Book"
},
"author": {
"#type": "Person",
"name": "Joe"
},
"reviewRating": {
"#type": "Rating",
"ratingValue": "7",
"bestRating": "10"
},
"publisher": {
"#type": "Organization",
"name": "Washington Times"
}
}
</script>
But according to this link I've to get review from a trusted review authority. I'm wondering why we need the structured data markup (where we have static 'rating', 'bestRating' etc value definitely these shouldn't be static) or how we can combine this with trusted review authority for getting dynamic ratting that changes over time?
If I'm understanding your question correctly, I think you are confusing two issues. Google requires reviews to be created using Schema markup in order for the review to have a chance to rank directly in the SERPs.
It is the companies that provide reviews: Yelp, Angie's List, Washington Times, etc, that have to format their content management systems to upload user generated review data into the proper markup.
So if you're a web developer working for one of these companies, then it makes sense to code the CMS so that the listings are displayed using schema markup.
If you are the marketer, your job is to get reviews, not format the way they are getting displayed.
There are of course other ways to use Schema markup on your own site to boost organic traffic. Consider for example the first SERP screenshot displayed in this article.
Here the webmaster has used schema markup to list three upcoming events in their result, which gives them four links in a single listing. This causes the listing to stand out and gives increased incentive for users to click, almost guaranteeing a higher click-thru rate than if they'd have not used the markup.
After reading several recent popular articles on the internet I decided to use ProfessionalService over LocalBusiness for my web design company. It is my understanding that LocalBusiness is very broad and it is best to be as specific as much as possible and the reason why I opted to use both ProfessionalService and additionalType with The Product Types Ontology.
Using Google Tag Manager my json-ld looks like this:
<script type="application/ld+json">
{
"#context": "http://schema.org",
"#type": "ProfessionalService",
"additionalType": "http://www.productontology.org/id/Web_design",
"name": "BYBE",
"url": "https://www.bybe.net",
"logo": "https://www.bybe.net/wp-content/themes/showboat/logo-bybe.png",
"description": "Creative website design company based in Bournemouth and Poole, Dorset.",
"telephone": "01202 949749",
"areaServed": ["Bournemouth", "Poole", "Dorset"],
"openingHoursSpecification": [
{
"#type": "OpeningHoursSpecification",
"dayOfWeek": [
"Monday",
"Tuesday",
"Wednesday",
"Thursday",
"Friday"
],
"opens": "09:00",
"closes": "17:00"
}
],
"address": {
"#type": "PostalAddress",
"streetAddress": "Flat 11, East Cliff Grange, 35 Knyveton Road",
"addressLocality": "Bournemouth",
"addressRegion": "Dorset",
"postalCode":"BH1 3QJ"
},
"geo": {
"#type": "GeoCoordinates",
"latitude": "50.73744",
"longitude": "-1.8495269"
},
"sameAs" : [ "https://plus.google.com/+ByBeBournemouth",
"https://twitter.com/bybe_net",
"https://www.facebook.com/ByBeUK",
"https://uk.pinterest.com/bybenet/",
"https://www.youtube.com/c/ByBeBournemouth",
"https://www.linkedin.com/company/bybe"]
}
</script>
I'm a little confused over the choice of words Schema has used on the ProfessionalService page:
SOURCE
Original definition: "provider of professional services."
The general ProfessionalService type for local businesses was
deprecated due to confusion with Service. For reference, the types
that it included were: Dentist, AccountingService, Attorney, Notary,
as well as types for several kinds of HomeAndConstructionBusiness:
Electrician, GeneralContractor, HousePainter, Locksmith, Plumber,
RoofingContractor. LegalService was introduced as a more inclusive
supertype of Attorney.
It's not clear if ProfessionalService is completely deprecated since it is still listed on the list of Schema's, I suspect they mean its deprecated for using it in a certain way, I'd be grateful if a Schema Jedi could shed some light on this issue.
Question(s):
Is ProfessionalService completely deprecated? If it's not then please include an example demonstrating the type of usage that is deprecated, that way it'll help and others.
ProfessionalService is deprecated for all cases, not only for some specific ones.
However, it will likely never be removed from Schema.org, because it would do more harm than good: many sites might still use this type, and many of them will probably never update their structured data (or even notice that it got deprecated in the meantime).
See also what the Schema.org webmaster, Dan Brickley, says about superseded types:
We shouldn't make the warnings too heavy or it creates awkwardness e.g. when search marketing people have recommended something to their clients then it gets superseded. We want consumers to respect older structures wherever possible and not worry publishers into constantly updating in the absence of concrete product-related incentives imho.
So if you have to use this type, nothing will break (just don’t expect updates for this type, or integration with future developments of the vocabulary). But if possible, it would better to use an alternative.
If not using ProfessionalService, the closest type for your web design company would be LocalBusiness. The services (design, development, consulting, CMS updates etc.) your company provides can be modelled with Service (where the provider is the LocalBusiness) and/or with makesOffer (where the Offer can reference the Service with itemOffered) (or with hasOfferCatalog in the same way, if you want to model it as list).
For school I am creating a deckbuilder website based on Magic the gathering. It's the project that decides if I get my degree or not. Trough the website from Deckbrew I have been able to get data like the following:
[
{
"name": "About Face",
"id": "about-face",
"url": "https://api.deckbrew.com/mtg/cards/about-face",
"store_url": "http://store.tcgplayer.com/magic/urzas-legacy/about-face",
"types": [
"instant"
],
"colors": [
"red"
],
"cmc": 1,
"cost": "{R}",
"text": "Switch target creature's power and toughness until end of turn.",
"formats": {
"commander": "legal",
"legacy": "legal",
"vintage": "legal"
},
"editions": [
{
"set": "Urza's Legacy",
"rarity": "common",
"artist": "Melissa A. Benson",
"multiverse_id": 12414,
"flavor": "The overconfident are the most vulnerable.",
"number": "73",
"layout": "normal",
"price": {
"low": 0,
"average": 0,
"high": 0
},
"url": "https://api.deckbrew.com/mtg/cards?multiverseid=12414",
"image_url": "http://mtgimage.com/multiverseid/12414.jpg",
"set_url": "https://api.deckbrew.com/mtg/sets/ULG",
"store_url": "http://store.tcgplayer.com/magic/urzas-legacy/about-face"
}
]
}
]
It's obvious that it's in jSon format. I have found the way to turn this into objects and the structure of the project is 4-layer MVC with entity framework and C#, which is working (kinda)...The problem is the database. I have been working on it for 2 months now and I am not getting any further. The thing I get stuck on is the database. I have not seen much on how to create databases and that's where it goes wrong. I don't get how to build the database. The creation itself would work if I figured out how to include certain things...
1) Formats: if the card is legal in a format, Formats is filled with: "legacy": "legal", "commander":"legal", ... so only the legal formats are included.
2) Types and colors are just plain arrays of words, but since I'm very bad with databases I don't even know how to figure this one out.
3) Editions is something completely different. It's an array of the object Edition which I believe has to have a table of its own. The problem here is that I thought I needed to use a foreign key but since it's an array of Editions I don't really know how to start doing that either.
4) and then there's Price: It always has 3 values: low, average and high which can be 0 if there's no price known.
So here you have it. To me this database is very complex or maybe I am making it too complex. Is there anybody who can help me to get this database organized so I can get on with my project, because I'm so lost at the moment that I feel I am not going to get this ready by the end of next month and that would be awful.
1: No, you should include all.
2: Table with colors, standard m:n binding table in between mapping the card table with the color table. Not knowing how to make a m:n relationship thing makes me thing you skipped all classes... this is fundamental and basic.
3: Seems like "cardedition" is the main table actually, and everything before is a master type table. Not sure- I don't really do magic at all, so I lack what is called domain knowledge. Are cards changed so multiple editions exist? Why is that an array in json?
3: magic values, 0,1,2,3. What is the question?
To me this database is very complex
I suggest you start from scratch (making things easier) and just have maybe 10 or so tables. Go step by step. Follow what you learned, go to 3rd of 4th normal form and go relational.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Is there any sort of API that just offers a simple symbol lookup service? i.e., input a company name and it will tell you the ticker symbol? I've tried just screen-scraping Google Finance, but after a little while it rate limits you and you have to enter a CAPTCHA. I'm trying to batch-lookup about 2000 ticker symbols. Any ideas?
You can use yahoo's symbol lookup like so:
http://d.yimg.com/autoc.finance.yahoo.com/autoc?query=yahoo&callback=YAHOO.Finance.SymbolSuggest.ssCallback
Where query is the company name.
You'll get something like this in return:
YAHOO.Finance.SymbolSuggest.ssCallback(
{
"ResultSet": {
"Query": "ya",
"Result": [
{
"symbol": "YHOO",
"name": "Yahoo! Inc.",
"exch": "NMS",
"type": "S",
"exchDisp": "NASDAQ"
},
{
"symbol": "AUY",
"name": "Yamana Gold, Inc.",
"exch": "NYQ",
"type": "S",
"exchDisp": "NYSE"
},
{
"symbol": "YZC",
"name": "Yanzhou Coal Mining Co. Ltd.",
"exch": "NYQ",
"type": "S",
"exchDisp": "NYSE"
},
{
"symbol": "YRI.TO",
"name": "YAMANA GOLD INC COM NPV",
"exch": "TOR",
"type": "S",
"exchDisp": "Toronto"
},
{
"symbol": "8046.TW",
"name": "NAN YA PRINTED CIR TWD10",
"exch": "TAI",
"type": "S",
"exchDisp": "Taiwan"
},
{
"symbol": "600319.SS",
"name": "WEIFANG YAXING CHE 'A'CNY1",
"exch": "SHH",
"type": "S",
"exchDisp": "Shanghai"
},
{
"symbol": "1991.HK",
"name": "TA YANG GROUP",
"exch": "HKG",
"type": "S",
"exchDisp": "Hong Kong"
},
{
"symbol": "1303.TW",
"name": "NAN YA PLASTIC TWD10",
"exch": "TAI",
"type": "S",
"exchDisp": "Taiwan"
},
{
"symbol": "0294.HK",
"name": "YANGTZEKIANG",
"exch": "HKG",
"type": "S",
"exchDisp": "Hong Kong"
},
{
"symbol": "YAVY",
"name": "Yadkin Valley Financial Corp.",
"exch": "NMS",
"type": "S",
"exchDisp": "NASDAQ"
}
]
}
}
)
Which is JSON and very easy to work with.
Hush... don't tell anybody.
Google Finance does let you retrieve up to 100 stock quotes at once using the following URL:
www.google.com/finance/info?infotype=infoquoteall&q=[ticker1],[ticker2],...,[tickern]
For example:
www.google.com/finance/info?infotype=infoquoteall&q=C,JPM,AIG
Someone has deciphered the available fields here:
http://qsb-mac.googlecode.com/svn/trunk/Vermilion/Modules/StockQuoter/StockQuoter.py
The current price ("l") is real-time and the delay is on par with Yahoo Finance. There are a few quirks you should be aware of. A handful of stocks require an exchange prefix. For example, if you query "BTIM", you'll get a "Bad Request" error but "AMEX:BTIM" works. A few stocks don't work even with the exchange prefix. For example, querying "FTWRD" and "NASDAQ:FTWRD" both generate "Bad Request" errors even though Google Finance does have information for this NASDAQ stock.
The "el" field, if present, tells you the current pre-market or after-hours price.
You can send an HTTP request to http://finance.yahoo.com requesting symbols, names, quotes, and all sorts of other data. Data is returned as a .CSV so you can request multiple symbols in one query.
So if you send:
http://finance.yahoo.com/d/quotes.csv?s=MSFT+F+ATT&f=sn
You'll get back something like:
"MSFT","Microsoft Corp"
"F","FORD MOTOR CO"
"ATT","AT&T"
Here is an article called Downloading Yahoo Data which includes the various tags used to request the data.
The NASDAQ site hosts separate CSV lists for ticker symbols in each stock exchange (NYSE, AMEX and NASDAQ). You need to complete the captcha and get the CSV dump.
http://www.nasdaq.com/screening/company-list.aspx
If you didn't want to sign up for a service, I'd probably go back to the exchanges themselves; most of them aren't CAPTCHAed yet...
The symbol lookup page for:
NYSE is at http://www.nyse.com/interface/html/SymbolLookup.html
NASDAQ is at http://www.nasdaq.com/asp/NasdaqSymLookup2.asp?mode=stock
London Stock Exchange is at http://www.londonstockexchange.com/en-gb/pricesnews/prices/Trigger/genericsearch.htm
ASX is at http://www.asx.com.au/asx/research/codeLookup.do
etc...
Use YQL and you don't need to worry. It's a query language by Yahoo and you can get all the stock data including the name of the company for the ticker. It's a REST API and it returns the results via XML or JSON. I have a full tutorial and source code on my site take a look: http://www.jarloo.com/yahoo-stock-symbol-lookup/
Currently, the NASDAQ web site publicly provides CSV files containing bulk listings -- it is broken up by first letter.
http://www.nasdaq.com/screening/companies-by-name.aspx?letter=A&render=download
Google Finance has an API - you probably have to apply for a developers key, but at least you'd save yourself the hassle of screen-scraping: http://code.google.com/apis/finance/reference.html
Your best bets are probably going with one of the other lookup services (still screen-scraping), and checking whether they don't require CAPTCHAs.
Yahoo Finance
MSN Money
AlphaTrade Finance
The last appears the least likely to require a CAPTCHA at any point, but it's worth checking all three.
Use YQL: a sql-like language to retrieve stuff from public api's:
YQL Console (external link)
It gives you a nice XML file to work with!
You can use the "Company Search" operation in the Company Fundamentals API here: http://www.mergent.com/servius/