Related
I have been learning coding for about a month now. I have some good experience with Python, and additionally I have completed this web development course on Udacity.
Now, I have a good foundation for programming, but one thing that confused me a lot is how to interact with various websites and APIs. The course I did briefly touched on this in terms of XML and JSON files and how some webpages offer their pages in these formats for easier reading by other machines.
But there are still a bunch of tasks which I have no idea how to approach whatsoever, but want to eventually do. I have constructed some hypothetical examples for the purpose of this question. I will post my current rough understanding of how I would do them below each one, and would appreciate feedback (on the API interaction, not on the front-end or on any back-end algorithms/AI/parsing):
Creating a phone application (disregarding the front-end part) which can then communicate with and perform rudimentary tasks on my computer.
I have no idea how to do this, and my guess would be that I would have to look into some external application/API meant for this process and implement this on both-ends of the system.
Being able to write a bot which goes on to a game website and controls the object via script. (e.g going onto a pacman game website written in flash and automatically controlling the character to avoid the ghosts)
I don't even know if this is possible, or how browser flash games interact handle the user-server interaction. Would I have to post some data via HTTP manually in the same way that playing in the keyboard would do? Or is everything done client side, in which case how would I fake user input? How would I get information on the ghost's position to work the AI?
Creating a mobile app for my school by allowing users to put their username and password into the app and then having the app automatically log in to the school and fetch certain data (e.g timetable) and return back in a readable form.
I'm guessing that I would take the input from the user on my mobile-app, and then navigate to the school's login page and POST this data in the relevant forms to log in. And then that I would (somehow, not sure), navigate to the timetable URL through my code while still managing to stay logged in, and then parse the html there?
I would appreciate some detail on how these kind of things are done, preferably with reference to these examples, so that I can get a better conceptual understanding.
Thanks!
Note: I have asked all those various questions mostly rhetorically, just so that those reading can get a better understanding of what my current programming level and understanding of APIs is at. I do not expect nor require specific answers for each and every question (so I hope this doesn't get flagged as being too vague or requiring too much detail!), I just would appreciate some responses telling me roughly how each of these APIs work approximately and how I would even start at looking at how to do these things.
You asked too many questions and honestly speaking I am not able to read and grasp entire text posted by you.
So, I am focusing only the title of your question:
"A conceptual understanding of API"
API (Application Programming Interface) means a set of functions which you can directly use by simply passing parameters to them.
Actually, in application development there are many common functions which every application programmer have to use. So, instead of coding them every time by every programmer, they are already coded in functions which you can use simply by passing parameters to them (if they need any external parameter).
Example:
I am offering you a maths API, set of functions {add, sub, mul, div}. You can pass two numbers to any of these four functions and get desired result instead of coding every time for ever operation like add, sub, mul and div.
Hope it helps...
I'm trying to test out my HIT on Mechanical Turk Sandbox with the categorization layout.
Everything works fine and I can deploy my HIT, except that I'm unable to figure out how to remove the requirement that workers must be Masters to participate.
Thus I am unable to complete the HIT through the Worker Sandbox because my account doesn't have a Masters qualification. I tried applying for the Masters qualification to no avail (and unsurprisingly so). I've found quite a few workarounds to removing the Masters qualification that were posted about back in 2011-2013 period, but it seems that Amazon has quite significantly changed the webpage since then.
Any tips would be greatly appreciated, or if there is another way for me to try out my own HIT on Sandbox.
Thomas' answer is the correct answer. If you create a Project using the Requester Website (http://requester.mturk.com/) and select either of "Sentiment" or "Categorization" template then it chooses Masters and there is no way of disabling it. It's worth noting that even though I describe these as "templates" they are in fact considered "applications" -- that is, they are designed to remove a lot of the complexity and "heavy lifting" that a Requester would normally do. That includes making approvals simpler (the "application" will take care of reviews and approvals), making pricing easier (the "application" will help recommend pricing based on marketplace conditions), and Worker quality (this is where Masters are selected by default and are non-optional).
Thomas' suggestion is to create your own template for MTurk Categorization (or Sentiment Analysis). This is probably the best suggestion. Check out "Tagging of an Image" for a template that you could quickly rework (assuming your categorization is of an image). You'll want to convert the three text inputs into a dropdown that includes your categorization. If you need to get more fancy (multi-level categorization) you can check out some fancy instantiations of it at sites like this:
http://bootsnipp.com/snippets/featured/multi-level-dropdown-menu-bs3
Sorry, as I'm sure that's not the answer you wanted to hear. But hope that helps lessen some of the time you're no-doubt spending to investigate this.
I've searched for a while, but I can't find anything related on Google or here.
Me and some friends were debating starting a company, so I figure it might be good to do a quick pilot project to see how well we can work together. We have a designer who can do HTML, CSS and Flash, enjoys doing art, but doesn't like to do HTML and CSS... And 2 programmers that are willing to do anything.
My question is, from an experienced site builder's perspective, what steps do we do - in chronological order - to properly handle a website? Does the designer design the look and feel of the site, then the programmers fill in the gaps with functionality? Or do the programmers create a "mock-up" of the site with most of the functionality, then the designer spices it up? Or is it more of a back-and-forth process?
I just want to know how a professional normally handles it.
Update:
A recap taking some of the notes from each post.
Step 1: Define requirements. What will your site/application do?
Step 2: Use cases. Who will use the application, and what will they do with it? This doesn't have to be done with a bunch of crazy UML diagrams, just use whatever visual aids you think work best for you. Find a CMS vendor, or a search vendor, or both. While planning, maybe do some competitor analysis, and see how those in similar fields have done theirs.
Step 3: Visual proof-of-concept. This is done by your designer, NOT your programmers... Programmers are notoriously bad at UI. Use an image program like Photoshop, not an HTML editor. Leave it fluid and simple at first. Select the three-color theme for the site (two primaries and an accent.) Get a sense of how you want to lay things out, keeping in mind the chosen CMS and/or search functionality. Focus hard on usability, add pizzaz later. Turn the created concept into JPEG mock-ups, or create a staging site to allow the client to view the work. A staging site will allow for future releases to be tested prior to moving it to production.
Step 4: Once the site is conceptualized by your designers, have your HTML/CSS developer turn it into markup. He/she should shoot for XHTML compliance and test on as many major browsers as you can. Also a good time to set up versioning/bug tracking/management systems, to keep track of changes, bugs, and feedback.
Step 5: Have your programmers start turning your requirements into software. This can and should be done in parallel with Step 4- there's no reason they can't be coding up the major pieces and writing tests while the UI is designed and developed.
Step 6: Marry up the final UI design with the code. Test, Test, Test!!
Step 7: Display end result to client, and get client sign-off.
Step 8: Deploy the site to production.
Rinse, Repeat...
Step 1: Define requirements. What will your site/application do?
Step 2: Use cases. Who will use the application, and what will they do with it? This doesn't have to be done with a bunch of crazy UML diagrams, just use whatever visual aids you think work best for you.
Step 3: Visual proof-of-concept. This is done by your designer, NOT your programmers. Use an image program like Photoshop, not an HTML editor. Leave it fluid and simple at first. Select the three-color theme for the site (two primaries and an accent.) Get a sense of how you want to lay things out. Focus hard on usability, add pizzaz later.
Step 4: Once the site is conceptualized by your designers, have your HTML/CSS developer turn it into markup. He/she should shoot for XHTML compliance and test on as many major browsers as you can.
Step 5: Have your programmers start turning your requirements into software. This can and should be done in parallel with Step 4- there's no reason they can't be coding up the major pieces and writing tests while the UI is designed and developed.
Step 6: Marry up the final UI design with the code. Test, Test, Test!!
Rinse, Repeat...
There is no one universal way. Every shop does it differently. Hence, a warning: gross generalizations follow.
Web development typically consists of much shorter release cycles, because it's so simple to push out a release, compared to client-side software. Thus the more "agile" methods are more frequently used than the "waterfall" models encountered in developing client software.
Figure out what, exactly, you're building.
Take care of all the legal stuff (e.g. what business entity you'll be forming, how will each team member be compensated for their work, will there be health benefits, etc).
Mockups. I suggest having the designers do the mockups since programmers are notoriously bad at UI design.
Set up some sort of bug tracking / case management system so that you have a centralized place for all your feature requests and bug reports.
Start coding.
Once you have a simple version of your app, get some people to test it out to make sure you're on the right path.
???
Profit!
As a first step, I'd recommend doing a bit of up-front design using an approach such as paper prototyping, to lock down what it is you want your website to do, and roughly how you want it to look.
Next up, read up on the Agile approach to software development and see if you like the sound of what it suggests. It tends to work best with smaller, well-motivated teams.
Figure out the minimum amount of functionality you can create that you can deliver as a product so that you can get user feedback as soon as possibly. Then expect to iteratively add functionality to the product over time.
The Web Style Guide provides a pretty detailed overview of the process.
You should mix and match the lists provided here for your needs.
I just want to make sure you know one thing...
Customers are "stoopid" when it comes to web design.
You will have to claw, scrape, drag, gnash, rip, and extricate every requirement from their naive little souls. If you fail to do so? Guess who gets the blame?
The road you now look down is a hard one filled with competition, stress, and risk. It requires endurance, faith, patience, and the ability to eat ramen 5 of 7 days a week.
To add (or repeat) Dave Swersky's list.
Gather requirements from clients
Do some competitor analysis. Gather
screen shots of competitor sites.
Build a sitemap /wireframe - What is
the structure/content of the site?
Get designers to create JPG mockups.
They may use the screen shots for
"inspiration"
Get feedback from
clients based on JPEG's
Create HTML
mockups from JPEG's
Get feedback
from clients. Go back to step 4 if
necessary
Implement HTML using
technology of choice
Unit test the site
UAT and obtain sign off.
Deploy to live
client feedback is critical, they should be involved in every step to ensure a successful implementation.
Hope this helps
In addition to the steps outlined in other answers, I'd add this (to be added somewhere near the end of the "cycle"):
x. Once you have a more or less end to end solution, set up a staging site.
y. Get client sign off on staging site.
z. Deploy to production site.
Celebrate! But not too hard, there's almost always going to be a few iterations of changes, because users rarely know exactly what they really want the first time around.
So, when (not if), the client asks for changes, you can work on the changes and promote them to the staging site first! This is important because a) it gives clients a chance to preview changes before the whole world sees them b) if the integrity of the data on the production site is important, you can hopefully weed out any issues on the staging site before they impact production data.
Just to give something on the other side of the coin. Where I work, we have for the past couple of years, worked on a redesign of the company's website. Here are some highlights of the process:
Identify vendors for various functions that will be needed. In this case that meant finding a Content Management System vendor as well as a Search vendor.
Get a new design for the site that can be applied to what was selected in the first step.
Using system integrators and in-house developers, start to build some of the functionality for the site and take the flexible, customizable software in 1 and make it useful for the organization. Note that this is where a couple of years have been spent getting this working and some business decisions ironed out.
Release a preview site to verify functionality and fix bugs, add enhancements as needed.
Note that in your case you may not have the same budget but there are various CMS frameworks out there to select as well as how much integration do you want to have for the site? Does it have to talk to a half-dozen different systems? In the case I mentioned above there are CRM integrations, ESB integrations, search integrations, and translation integrations to give a few examples of where things had to be wired up correctly.
In response to the comment, be sure you and the client know what is meant by "simple" as if there is any e-commerce functionality, forums, or personalization these are examples where it can be important to know what is needed now and have an idea of what is needed down the road as there can likely be a ton of things that customers may want but you have to figure out some of the nitty-gritty details at points in the future. For example, some people may think that Google is simple, and from an end-user perspective it is though how many computers does Google have running how many different applications doing how much processing 24/7? Quite a bit, I'd imagine. Simple is good, but sometimes making something look simple can be incredibly hard to do.
One of our next projects is supposed to be a MS Windows based game (written in C#, with a winform GUI and an integrated DirectX display-control) for a customer who wants to give away prizes to the best players. This project is meant to run for a couple of years, with championships, ladders, tournaments, player vs. player-action and so on.
One of the main concerns here is cheating, as a player would benefit dramatically if he was able to - for instance - let a custom made bot play the game for him (more in terms of strategy-decisions than in terms of playing many hours).
So my question is: what technical possibilites do we have to detect bot activity? We can of course track the number of hours played, analyze strategies to detect anomalies and so on, but as far as this question is concerned, I would be more interested in knowing details like
how to detect if another application makes periodical screenshots?
how to detect if another application scans our process memory?
what are good ways to determine whether user input (mouse movement, keyboard input) is human-generated and not automated?
is it possible to detect if another application requests informations about controls in our application (position of controls etc)?
what other ways exist in which a cheater could gather informations about the current game state, feed those to a bot and send the determined actions back to the client?
Your feedback is highly appreciated!
I wrote d2botnet, a .net diablo 2 automation engine a while back, and something you can add to your list of things to watch out for are malformed /invalid/forged packets. I assume this game will communicate over TCP. Packet sniffing and forging are usually the first way games (online anyways) are automated. I know blizzard would detect malformed packets, somehting i tried to stay away from doing in d2botnet.
So make sure you detect invalid packets. Encrypt them. Hash them. do somethign to make sure they are valid. If you think about it, if someone can know exactly what every packet means that is sent back and forth they dont even need to run the client software, which then makes any process based detection a moot point. So you can also add in some sort of packet based challenge response that your cleint must know how to respond to.
Just an idea what if the 'cheater' runs your software in a virtual machine (like vmware) and makes screenshots of that window? I doubt you can defend against that.
You obviously can't defend against the 'analog gap', e.g. the cheater's system makes external screenshots with a high quality camera - I guess it's only a theoretical issue.
Maybe you should investigate chess sites. There is a lot of money in chess, they don't like bots either - maybe they have come up with a solution already.
The best protection against automation is to not have tasks that require grinding.
That being said, the best way to detect automation is to actively engage the user and require periodic CAPTCHA-like tests (except without the image and so forth). I'd recommend utilizing a database of several thousand simple one-off questions that get posed to the user every so often.
However, based on your question, I'd say your best bet is to not implement the anti-automation features in C#. You stand very little chance of detecting well-written hacks/bots from within managed code, especially when all the hacker has to do is simply go into ring0 to avoid detection via any standard method. I'd recommend a Warden-like approach (download-able module that you can update whenever you feel like) combined with a Kernel-Mode Driver that hooks all of the windows API functions and watches them for "inappropriate" calls. Note, however, that you're going to run into a lot of false positives, so you need to not base your banning system on your automated data. Always have a human look over it before banning.
A common method of listening to keyboard and mouse input in an application is setting a windows hook using SetWindowsHookEx.
Vendors usually try to protect their software during installation so that hacker won't automate and crack/find a serial for their application.
Google the term: "Key Loggers"...
Here's an article that describes the problem and methods to prevent it.
I have no deeper understanding on how PunkBuster and such softwar works, but this is the way I'd go:
Iintercept calls to the API functions that handle the memory stuff like ReadProcessMemory, WriteProcessMemory and so on.
You'd detect if your process is involved in the call, log it, and trampoline the call back to the original function.
This should work for the screenshot taking too, but you might want to intercept the BitBlt function.
Here's a basic tutorial concerning the function interception:
Intercepting System API Calls
You should look into what goes into Punkbuster, Valve Anti-Cheat, and some other anti-cheat stuff for some pointers.
Edit: What I mean is, look into how they do it; how they detect that stuff.
I don't know the technical details, but Intenet Chess Club's BlitzIn program seems to have integrated program switching detection. That's of course for detecting people running a chess engine on the side and not directly applicable to your case, but you may be able to extrapolate the apporach to something like if process X takes more than Z% CPU time the next Y cycles, it's probably a bot running.
That in addition to a "you must not run anything else while playing the game to be eligible for prizes" as part of the contest rules might work.
Also, a draconian "we might decide in any time for any reason that you have been using a bot and disqualify you" rule also helps with the heuristic approach above (used in prized ICC chess tournaments).
All these questions are easily solved by the rule 1 above:
* how to detect if another application makes periodical screenshots?
* how to detect if another application scans our process memory?
* what are good ways to determine whether user input (mouse movement, keyboard input) is human-generated and not automated?
* is it possible to detect if another application requests informations about controls in our application (position of controls etc)?
I think a good way to make harder the problem to the crackers is to have the only authoritative copies of the game state in your servers, only sending to and receiving updates from the clients, that way you can embed in the communication protocol itself client validation (that it hasn't been cracked and thus the detection rules are still in place). That, and actively monitoring for new weird behavior found might get you close to where you want to be.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I work on a complex application where different teams work on their own modules with a degree of overlap. A while back we got a Mediawiki instance set up, partly at my prompting. I have a hard job getting people to actually use it, let alone contribute.
I can see a lot of benefit in sharing information. It may at least reduce the times we reinvent the wheel.
The wiki is not very structured, but I'm not sure that is a problem as long as you can search for what you need.
Any hints?
Some tips:
Any time someone sends information by email that really should be in a wiki, make a page for that topic and add what they put in the email. Then reply "Thanks for that info, I've put it into the wiki here so that it's easier to find in the future."
Likewise, if you have information you need to share that should be in the wiki, put it there and just send an email with a link to it, rather than email people.
When you ask people for information, phrase it so that putting such documentation in the wiki should be considered the default or standard: "I searched in the wiki but I couldn't find it. Have you put that info up there yet?"
If you are the "wiki champion", make sure other people know how to use it, e.g. "Did I go through how to create a new page with you yet?"
Edit the sidebar to make sure it is relevant to your work.
Use "nav box" style templates on related pages for easier navigation.
Put something like {{Special:NewPages/5}} on the front page, or recent changes, so that people can see the activity.
Take a peek at Recent changes every few days or week, and if you notice someone adding information without being prodded, send them an email or drop by and give them a little compliment.
As I mentioned before, a Wiki is very unorganized.
However, if that is the only argument from your developers, then invest some effort to create a simple index page and keep it updated (either do it yourself or ask people to link their contributions to the index). That way, the Wiki might grow into a very nice and quite comprehensive collection of documentation for all your work.
We've been using a wiki in some form or another for a while now, but it does take a while for people to get on board. You might find that you will be the only one writing articles for some time, but bear with it, other people will come on board eventually.
If someone sends an email around that contains information related to the project then helpfully point them in the direction of the wiki - and keep doing that - they should get the hint.
We have a SharePoint portal and use the wiki from there - we customised it with our own branding so that it "looks the part" - I really feel this has helped to improve the uptake of it.
Make sure that everyone is aware that the wiki is even more informal than email.... because there will be a "fear factor" that people may think anything they add to the wiki will be over-analysed.
I think most of the answers so far are spot on - the more you plug away at it yourself, the larger the body of useful information will become, so slowly but surely people will naturally start to use it.
The other approach you could use is this: Suggest that every time someone asks another team member a question about the project, they should answer the question as normal, but also add the answer to a section of the Wiki. This may take a few minutes extra, but it will mean that the next time someone asks the same question (which they inevitably will), you can save time by pointing them at the Wiki. This, in turn, should help people to start using the Wiki as a first source of information and help overall up-take.
You can't force developers to do something they do not have an incentive of using for; unfortunately wikis, like documentation (well, in fact wikis are documentation) rarely have any "cool" value for developers. Besides, they're already deep into dev work -- could you really bother them with a wiki?
That being said, the people who pushed for the wiki (e.g., you) should be primarily responsible for updating it, and you really would have a lot of work cut out for you if you're serious about it.
You might also try the ff:
It's not very structured you say -- a lot of people get turned off from ill-structured (hard-to-search/browse) wikis. So maybe you can fix that first
Maybe you can ask lead developers/project managers to populate it with things that are issues for them: things like code conventions and API design for your particular project
Lead by example: religiously document your part of the system. Setting a precedent may encourage others to do the same
Sell the idea of using the wiki to the developers. You've identified some benefits, share those with the developers. If they can see that they'll get something of value out of it they'll start using it.
Example advantages from What Is a Wiki
Good for writing down quick ideas or longer ones, giving you more time for formal writing and editing.
Instantly collaborative without emailing documents, keeping the group in sync.
Accessible from anywhere with a web connection (if you don't mind writing in web-browser text forms).
Your archive, because every page revision is kept.
Exciting, immediate, and empowering--everyone has a say.
I have done some selling and even run some training sessions. I think some people are turned off by the lack of WYSIWYG editing and ability to paste formatted text from Word or Outlook. I know there are some tools to work around these, but they are still barriers.
There are some areas where the wiki is being used to log certain areas, but people who update those are not doing anything else with it.
I will use the wiki to document my specialised area regardless as it acts as a convenient brain extension. When starting a new development I use it as a notepad for ideas that I can expand on as it progresses.
It would help if management would give it some vocal support, even if it is not made mandatory.
I have a hard job getting people to actually use it, let alone contribute.
One of the easiest ways to get people to contribute to a wiki, is to actually have them provide contents in a wiki-suitable fashion, i.e. so that whatever they post using their usual channels of communications (newsgroups, mailing lists, forums, issue trackers, chat), is basically suitable for inclusion on the wiki.
So that others (users/volunteers) can simply take such contents and put them on the wiki.
This sounds more complicated than it really is, it's mostly about generalizing questions and answers, so that they are not necessarily part of a conversation, but can be comprehensible, meaningful and useful in a standalone fashion.
For example a question like the following:
how do I get git to clone a remote repository???
Can be answered like this:
Hello,
Just use git clone git://...
But questions can also be answered in a less personal style:
In order to clone a git repository, you will want to use the clone parameter to git:
git clone git://....
What I am trying to say is that most discussions in a project can and should be easily used to become documentation eventually. With this sort of mindset, your documentation can actually grow rather rapidly. You only need to get people to keep in mind that useful information should be ideally provided in a fashion that is suitable for wiki inclusion.
I have witnessed several instances where open source projects started to use this approach to some extent and while some people (largely new users) complained that answers were not very personal, the body of documentation was increasing steadily, because other people simply monitored such discussions and started to copy/paste such responses to the wiki.
Basically, this is one of the easiest ways to get people to contribute to a wiki, without requiring them to actually use it themselves, the only thing that's required of them is a shift in thinking.
If the developers still need to maintain 'real' documentation (s.a. Word documents), I see no way to meaningfully duplicate that on a Wiki.
It does not make sense for people to write twice
Any duplicated data is prone to get out of sync, soon.
What my current customer has done is move all this to Wiki. So I only document once, and I do it on the Wiki.
This is okay. Working with Wiki is more tedious than with Word, but at least the doc is online and others can mix-and-match with it.
Another working solution (imho) would be to store docs alongside the source, on subversion. But then the merging system needs to be able to cope with rich text etc. as well. I don't know, if any solution for that exists (other than using HTML or LaTex, which actually would not be bad picks).
Find "sticky" items (sub-3 pg. docs / diagrams / etc) something that the team seems to be creating again and again & post it on the wiki. Make sure everyone has access to the wiki and knows its there - set up a notification mechanism if possible. With some luck, the next time they have to access, rather than dig it out of version control or their machines - they should hit the wiki.
If they still don't, try to see if the team has enough slack to actually use the wiki - Subtler issues may lie beneath their reluctance.
Take a look at the advice at http://www.ikiw.org/ Grow your Wiki
Just to add to some of the excellent advice being offered here...
As a dev in a small company that does largely gov't contract work in the 6-24 month range, I find that my time is often split between development and writing status reports (right up there with writing documentation, only worse!) Having a wiki to slap down unorganized thoughts and notes as we go along has made report-writing a lot less painful (not pain-LESS, but better all the same).
Further, if you're already in the Mediawiki world, you might want to look at SemanticMediawiki. It allows you to take the organization of your data to another level by semantically tagging it. That doesn't mean a lot on its own, I know, but I can tell you (for example) that it can drastically improve the relevance of the data returned from searches. It is definitely worth a look.
Generally good advice here. I'd like to add:
You really need a champion - someone pushing this to developers and management (without being pushy - that's a challenge!) and providing support & tutorials when possible. This person also needs to be a peer (so a fellow developer, not someone in a remote IT department) and really customer focused i.e. ready to make changes when requested.
Speaking of changes, some people here say wikis are unstructured. I disagree. Our MediaWiki installation is structured using categories, particularly with two extensions:WarnNoCategories (to require users to add a category when saving a page) and CategoryTree to show how all the categories fit together (this can be linked to from the sidebar). I've got more tips on how we keep this low threshold, if you're interested.