I am developing a sensor based mobile application for iOS and Android. The data produced by smart phone sensors will be stored in the cloud. At this point, I am wondering that what I should test about the data transfer and storing. I mean that for example, I should test the scenario as if the connection corrupts while GPS data transfer not finished. I am not looking for the techniques, or testing styles. I am trying to find possible failure points or test scenarios. I hope that I could explain my point.
Below are some of the things worth considering for your app:
Incomplete transfers when connection corrupts (as u mentioned)
Cloud-server size..how much request can it handle at a single instance?
If u are considering cloud solutions, you should also consider the location of your users from where they will be accessing your app. Users and the location of data center will also affect in the response time.
Format of the date to stored. Considering a file size which is fast in i/o will also help optimize the speed of the app.
Asynchronous/Synchronous data transfer
Security measures on the cloud..may be using services like VPC if you are considering AWS
These are some things worth considering.
Thanks :)
Related
I'm creating a RESTful backend API for eventual use by a phone app, and am toying with the idea of making some of the API read functions nothing more than static files, created and periodically updated by my server-side code, that the app will simply GET directly.
Is this a good idea?
My hope is to significantly reduce the CPU and memory load on the server by not requiring any code to run at all for many of the API calls. However, there could potentially be a huge number of these files (at least one per user of the phone app, which will be a public app listed in the app stores that I naturally hope will get lots of downloads) and I'm wondering if that alone will lead to latency issues I'm trying to avoid.
Here are more details:
It's an Apache server
The hardware is a hosting provider's VPS with about 1gb memory and 20gb free disk space
The average file size (in terms of content and not disk footprint) will probably be < 1kb
I imagine my server-side code might update a given user's data once a day or so at most.
The app will probably do GETs on these files just a few times a day. (There's no real-time interaction going on.)
I might password protect the directory the files will be in at the .htaccess level, though there's no personal or proprietary information in any of the files, so maybe I don't need to, but if I do, will that make a difference in terms of the main question of feasibility and performance?
Thanks for any help you can give me.
This is generally a good thing to do: anything that can be static rather than dynamic is a win for performance and cost (it's why we do caching!), but the main issue with with authorization (which you'll still need to do for each incoming request).
You might also want to consider using a cloud service for storage of the static data (e.g., Amazon S3 or Google Cloud Storage). There are neat ways to provide temporary authorized URLs that you can pass to users so that they can read the data for a short time and then must re-authorize to continue having access.
I am trying to do something new, something I have never done before. I am looking for advice or point me into right direction how to choose technology. I am trying to build race simulation app that will have thousands of iot devices streaming data into central platform. While I understand that I can use some sort of IOT hub with cloud providers, but what technology do I choose for storing data?
Example is online indoor biking app. There are apps where you can connect your indoor bike online and have simulated race. For my project I am trying to build something similar. Do I use NO SQL db in this scenario? What technology will allow better scale of application like this since it could be millions of devices around the world in "simulated" race. I am not worried about front-end and things like that, but backend, IOT hub, storing data, presenting-real time?
At this point it is important to understand what kind of data your IoT devices will stream, and at what kind of a rate. It will have significant impact on your question.
That it is if it's just location information and some other small data sent lets say once a second, then if you're talking about tens of thousands of devices - this is not a big load of information, and any standard database, like MySQL will be able to deal with it. You will of course need a multi-threaded server(s) capable of handling many requests in parallel.
If your IoT devices will stream HD video, then you're looking at a completely different solution, with a much stronger server, capable of handling allot of streams in parallel, with significant bandwidth requirements from your hosting company, as well as storage space for all the videos. In this case you will store the streams as files (if you'll need them later on), and you won't need any special database either.
In any case, once you'll reach millions of users, you'll be able to scale most modern databases and servers, like MySQL replication capability. For example, take a look how Wikipedia is relying on MySQL: wikipedia - MySQL https://www.mysql.com/why-mysql/case-studies/mysql-cs-wikipedia.html
So I wouldn't be worried regarding the database on this stage, but make sure that the design of my system is in accordance to the the type of data and rate it is streamed.
Hope this gives you a pointer.
I'm developing an app that allows to track a mobile device instantly (live) ... I need an of advice. The application must send the location to a webservice that in it's turn records the received data in a database.
What would be, in your opinion, the best way to store the location values?
I'm new in using bigdata and I'm afraid that simple sql requests wont be able to do the work properly ... I imagine if there is lot of users and each user send a request each 1sec I'll have issue with the database ...
An advice ? Thank you very much
i think you could have a look into the geospatial queries in mongo, if you chose to go ahead with mongodb.
Refer here
And here
for the design of the database would depend on the nature of the query (essentially the read and write).
Worth having a look into
Working at Cintric we landed on using elasticsearch. We process billions of location points in real time and provide advanced analytics to our users.
We started with mongoDB and ran into a lot of troubles, eventually leading to a painful migration.
Our stack currently has mobile devices dump location updates into AWS Kinesis, which are then processed by AWS Lambda handlers, and then dumped into elasticsearch. We're able to serve, process and store 300 million requests/month for only a few hundred dollars/month. Analytics for our dashboard add additional cost but for your needs I would highly recommend checking out your options on AWS.
I am running an Azure web role, which is storing very small blobs into Azure storage. (Blob upload is being done from the server, not from the browser.) I have searched stack overflow and the rest of the internet for tips on optimizing blob storage performance, and I believe I've checked and implemented all of the usual suspects: uploading async, allowing unlimited outgoing web connections (which now seems to be the default setting on web roles and no longer needs to be explicitly set in web.config or in code).
Tweaking the number of concurrent uploads I allow makes some difference, but regardless of what I've tried, I seem to max out at around 1,000 blob uploads per second. This is when running in the Azure web role, in the same region as the storage account (East US). My rate when running this from home over a good internet connection isn't much less, ~700 blobs/sec, which seems to tell me that it's not the network latency that's limiting the rate, it's the actual processing time of the storage service.
I wouldn't normally consider these rates horrible for this kind of a service, but I've read that Microsoft boasts a rate of ~20,000 storage transactions per second, so I've been a little disappointed with these results.
I'd like to get some feedback from those who have really tried to push the limits of blob storage. Does ~1000 small uploads per second sound about right? Or is there possibly something else I should be doing to improve this? I'll post the code if I need to, but I'd rather not receive speculative answers, I'd like to hear from developers who can either confirm that my results are reasonable, or that they've seen much higher throughput.
I should add that I'm currently running this in a small web role. I've tried it also in a medium web role, and didn't see any significant difference.
EDIT:
After a few days of development and testing, my upload rate seemed to suddenly increase. Not by a lot, but maybe by another ~200 per second. In looking around the web, I noticed a comment in the Azure documentation stating "A storage account scales automatically as usage increases." So I'm wondering if it really is capable of much higher rates, but will not automatically scale up until it sees sustained period of high volume. Some confirmation of that would also be greatly appreciated.
Depending on how small your requests are the problem might be caused by Nagle’s Algorithm is Not Friendly towards Small Requests - although usually I see that with queues / table operations. Try disabling Nagle's and let me know if that makes any difference. As an fyi, you have to disable it prior to establishing the connection otherwise the changes will not take effect.
Jason
I would like to create an app where the user can add and view data. Either adding at a desktop/tablet or phone and reading from either source. I would like the data store to be synced between any of the user's devices.
I'm starting to play with the Trial of Azure, and it looks promising. Probably a solid way to sync data through to cloud between users' devices. Other than syncing between a users devices, I have no need for cloud services currently.
I've seen some apps that do a 'Backup/Restore' model with the user's SkyDrive account. But this seems to be a manual process. I'd like to see it done seamlessly.
I've looked into Sync services, but that would be more like a hub/spoke solution. Again, I don't need a central database.
What are some options? At this point, I would be fine using just Windows 8 patterns/practices.
Because they are separate devices, you will need to have some service layer to do the store/forward for you. With that you have two basic choices, use the end user's own storage (aka SkyDrive) or use your own storage (aka Windows Azure).
SkyDrive is fully supported through the Live SDKs and provides a secure way to allow a user to share store their data, and synchronize it across multiple devices. You get security, and there is no cost for the server side storage on your part. The user owns their storage, not you. The limitation is that you may have issues sharing that same data across other devices or users where SkyDrive (or whatever file sync service you use) is not available.
With a service layer, like Azure, you have a lot more flexibility, but you also will be responsible for maintaining (and paying for) that server side storage / services. Have you looked at "Windows Azure Mobile Services". With your Azure account you get 10 free Azure Mobile Services. You will pay for the SQL data storage on the backend, and that cost will depend on the amount of data you store on the server side. You also need to make sure to architect your application in a way to protect an individual users' data, but it is actually pretty easy to do, well documented, and gives you a lot of options.
Lastly, you may consider what type of data you want to share. SkyDrive is great for "Files". Pics, Songs, Videos, etc. Windows Azure Mobile Services (WAMS) is great for "Data".
Neither model is right or wrong. It just depends on your goals.
Hope that helps you go through the thought process