Hi I am new to Mechanical Turk. I have 10k images and I want to ask turkers to write down a short summary for each image in Mechanical Turk. Since all images in my image set are similar, when a Turker does the similar summarization task more than 10 times, he'll find out some tricks in this task and write down similar summary of the following images.
To increase the diversity and randomness, I want to ask as many different people to do the task as possible. The perfect strategy is that one unique turker is only allowed to label just one image (or less than 10 images), while one image can be summarized by more than one turker. My experiment aims at collect different textual summarization from different people which covers a rich vocabulary set.
If I understand you correctly, you have unique images in total to label. It sounds like you're looking to have each task (HIT) request that a Worker label 10 unique images. This will result in 1K HITs with 10 images per HIT. If you'd like one just one unique Worker to label each image, then you'll set the Number of Assignments Requested to 1. If you'd like for multiple Workers to work on the same image (say, to ensure quality, or just to broaden the number and type of labels you might get) then you'll set Number of Assignments Requested to the # of Unique Workers you'd like to work on each task (HIT).
If I've misunderstood what you're looking to do, just clarify and I'll be happy to revise my answer.
You can learn more about these concepts and more here:
http://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/mechanical-turk-concepts.html
Good luck!
Yes, this is possible. According to documentation:
http://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkRequester/amt-dg.pdf (page 4)
You can specify the maximum number of assignments that any Worker can accept for your HITs.You
can set two types of limits:
The maximum number of assignments any Worker can accept for a specific HIT type you've created
The maximum number of assignments any Worker can accept for all your HITs that don't otherwise
have a HIT-type-specific limit already assigned
Related
I have a domain model where I penalize a score at multiple levels within a single rule. Consider a cloud scheduling problem where we have to assign processes to computers, but each process can be split amongst several computers. Each process has a threshold (e.g. 75%), and we can only "win" the process if we can schedule up to its threshold. We get some small additional benefit from scheduling the remaining 25% of the process, but our solver is geared to "winning" as many processes as possible, so we should be scheduling as many processes as possible to their threshold before scheduling the remainder of the process.
Our hard rule counts hard constraints (we can't schedule more processes on a computer than it can handle)
Our medium rule is rewarded for how many processes have been scheduled up to the threshold (no additional reward for going above 75%).
Our soft rule is rewarded for how many processes have been scheduled total (here we do get additional reward for going above 75%).
This scoring implementation means that it is more important to schedule all processes up to their threshold than to waste precious computer space scheduling 100% of a process.
When we used a drools implementation, we had a rule which rewarded the medium and soft levels simultaneously.
when
$process : Process()
$percentAllocated : calculatePercentProcessAllocated($process) //uses an accumulator over all computers
then
mediumReward;
if ($percentAllocated > $process.getThreshold()) {
mediumReward = $process.getThreshold();
}
else {
mediumReward = $percentAllocated;
}
softReward = $percentAllocated;
scoreHolder.addMultiConstraintMatch(0, mediumReward, softReward);
The above pseudo-drools is heavily simplified, just want to show how we were rewarded two levels at once.
The problem is that I don't see any good way to apply multi constraint matches using constraint streams. All the examples I see automatically add a terminator after applying a penalize or reward method, so that no further score modifications can be made. The only way I see to implement my rules is to make two rules which are identical outside of their reward calls.
I would very much like to avoid running the same constraint twice if possible, so is there a way to penalize the score at multiple levels at once?
Also to anticipate a possible answer to this question, it is not possible to split our domain model so that each process is two processes (one process from 0% to the threshold, and another from the threshold to 100%). Part of the accumulation that I have glossed over involves linking the two parts and would be too expensive to perform if they were separate objects.
There is currently no way in the constraint streams API to do that. Constraint weights are assumed to be constant.
If you must do this, you can take advantage of the fact that there really is no "medium" score. The medium part in HardMediumSoftScore is just another level of soft. Therefore 0hard/1medium/2soft would in practice behave the same as 0hard/1000002soft. If you pick a sufficiently high constant to multiply the medium part with, you can have HardSoftScore work just like HardMediumSoftScore and implement your use case at the same time.
Is it a hack? Yes. Do I like it? No. But it does solve your issue.
Another way to do that would be to take advantage of node sharing. I'll use an example that will show it better than a thousand words:
UniConstraintStream<Person> stream = constraintFactory.forEach(Person.class);
Constraint one = stream.penalize(HardSoftScore.ONE).asConstraint("Constraint 1")
Constraint two = stream.penalize(HardSoftScore.TEN).asConstraint("Constraint 2")
This looks like two constraints executing twice. And in CS-Drools, it is exactly that. But CS-Bavet can actually optimize this and will only execute the stream once, applying two different penalties at the end.
This is not a very nice programming model, you lose the fluency of the API and you need to switch to the non-default CS implementation. But again - if you absolutely need to do what you want to do, this would be another way.
I'm trying to learn OptaPlanner by building a playlist generator. My constraints look roughly like:
The total time must be ~60 minutes.
All songs selected must be unique.
Each ~15 minute block may contain only songs from a single artist.
The four 15 minute blocks must be different artists.
So by hierarchical, I mean you could choose the artist and then attempt to fill the block.
My current implementation expresses these as constraints on song selection. It's able to solve the problem, but I feel like it's spending a lot of time trying to align the artist constraint.
Reading through docs, it seems like there are some features which might be helpful:
Partitioned search
Chained variables
Custom move selectors (e.g. changing all the songs to have consistent artist)
Different weighting on constraints
What's the recommended way to handle this type of relationship?
Sounds like these are just 4 constraints with 4 different score levels (see docs chapter on score calculation). So in that case, you'll need a BendableScore.
In practice though, I would be surprised if 3. and 4. are hierarchical: 3 might just have a much heavier score weight than 4. In that case a HardMediumSoftScore suffices.
I am currently developing a neural network that tries to make a suggestion for a specific user based on his recent activities. I will try to illustrate my problem with an example.
Now, let's say im trying to suggest new music to a user based on the music he recently listened to. Since people often listen to artists they know, one input of such a neural network might be the artists he recently listened to.
The problem is the encoding of this feature. As the id of the artist in the database has no meaning for the neural network, the only other option that comes to my mind would be one-hot encoding every artist, but that doesn't sound to promising either regarding the thousands of different artists out there.
My question is: How can i encode such a feature?
The approach you describe is called content-based filtering. The intuition is to recommend items to customer A similar to previous items liked by A. An advantage to this approach is that you only need data about one user, which tends to result in a "personalized" approach for recommendation. But some disadvantages include the construction of features (the problem you're dealing with now), the difficulty to build an interesting profile for new users, plus it will also never recommend items outside a user's content profile. As for the difficulty of representation, features are usually handcrafted and abstracted afterwards. For music specifically, features would be things like 'artist', 'genre', etc. and abstraction for informative keywords (if necessary) is widely done using tf-idf.
This may go outside the scope of the question, but I think it is also worth mentioning an alternative approach to this: collaborative filtering. Rather than similar items, here we instead try to find users with similar tastes and recommend products that they liked. The only data you need here are some sort of user ratings or values of how much they (dis)liked some data - eliminating the need for feature design. Furthermore, since we analyze similar persons rather than items for recommendation, this approach tends to also work well for new users. The general flow for collaborative filtering looks like:
Measure similarity between user of interest and all other users
(optional) Select a smaller subset consisting of most similar users
Predict ratings as a weighted combination of "nearest neighbors"
Return the highest rated items
A popular approach for the similarity weighting in the algorithm is based on the Pearson correlation coefficient.
Finally, something to consider here is the need for performance/scalability: calculating pairwise similarities for millions of users is not really light-weight on a normal computer.
I'm working on a very high throughput site with many items, am looking into implementing "trending now" type functionality, that would allow users to quickly get a prioritized list of the top N items that have been viewed recently by many people, that gradually fade away as they get fewer views.
One idea about how to do this is to give more weight to recent views of an item, something like a weight of 16 for every view of an item the past 15 minutes, a weight of 8 for every view of an item in the past 1 hour, a weight of 4 for things in the past 4 hours, etc but I do not know if this is the right way to approach it.
I'd like to do this in Redis, we've had good success with Redis in the past for other projects.
What is the best way to do this, both technologically and the determination of what is trending?
The first answer hints at a solution but I'm looking for more detail -- starting a bounty.
These are both decent ideas, but not quite detailed enough. One got half the bounty but leaving the question open.
So, I would start with a basic time ordering (zset of item_id scored by timestamp, for example), and then float things up based on interactions. So you might decided that a single interaction is worth 10 minutes of 'freshness', so each interaction adds that much time to the score of the relevant item. If all interactions are valued equally, you can do this with one zset and just increment the scores as interactions occur.
If you want to have some kind of back-off, say, scoring by the square root of the interaction count instead of the interaction count directly, you could build a second zset with your score for interactions, and use zunionstore to combine this with your timestamp index. For this, you'll probably want to pull out the existing score, do some math on it and put a new score over it (zadd will let you overwrite a score)
The zunionstore is potentially expensive, and for sufficiently large sets even the zadd/zincrby gets expensive. To this end, you might want to keep only the N highest scoring items, for N=10,000 say, depending on your application needs.
These two links are very helpful:
http://stdout.heyzap.com/2013/04/08/surfacing-interesting-content/
http://word.bitly.com/post/41284219720/forget-table
The Reddit Ranking algorithm does a pretty good job of what you describe. A good write up here that talks through how it works.
https://medium.com/hacking-and-gonzo/how-reddit-ranking-algorithms-work-ef111e33d0d9
consider an ordered set with the number of views as the scores. whenever an item is accessed, increment its score (http://redis.io/commands/zincrby). this way you can get top items out of the set ordered by scores.
you will need to "fade" the items too, maybe with an external process that would decrement the scores.
Tried doing a bit of research on the following with no luck. Thought I'd ask here in case someone has come across it before.
I help a volunteer-run radio station with their technology needs. One of the main things that have come up is they would like to schedule their advertising programmatically.
There are a lot of neat and complex rule engines out there for advertising, but all we need is something pretty simple (along with any experience that's worth thinking about).
I would like to write something in SQL if possible to deal with these entities. Ideally if someone has written something like this for other advertising mediums (web, etc.,) it would be really helpful.
Entities:
Ads (consisting of a category, # of plays per day, start date, end date or permanent play)
Ad Category (Restaurant, Health, Food store, etc.)
To over-simplify the problem, this will be a elegant sql statement. Getting there... :)
I would like to be able to generate a playlist per day using the above two entities where:
No two ads in the same category are played within x number of ads of each other.
(nice to have) high promotion ads can be pushed
At this time, there are no "ad slots" to fill. There is no "time of day" considerations.
We queue up the ads for the day and go through them between songs/shows, etc. We know how many per hour we have to fill, etc.
Any thoughts/ideas/links/examples? I'm going to keep on looking and hopefully come across something instead of learning it the long way.
Very interesting question, SMO. Right now it looks like a constraint programming problem because you aren't looking for an optimal solution, just one that satisfies all the constraints you have specified. In response to those who wanted to close the question, I'd say they need to check out constraint programming a bit. It's far closer to stackoverflow that any operations research sites.
Look into constraint programming and scheduling - I'll bet you'll find an analogous problem toot sweet !
Keep us posted on your progress, please.
Ignoring the T-SQL request for the moment since that's unlikely to be the best language to write this in ...
One of my favorites approaches to tough 'layout' problems like this is Simulated Annealing. It's a good approach because you don't need to think HOW to solve the actual problem: all you define is a measure of how good the current layout is (a score if you will) and then you allow random changes that either increase or decrease that score. Over many iterations you gradually reduce the probability of moving to a worse score. This 'simulated annealing' approach reduces the probability of getting stuck in a local minimum.
So in your case the scoring function for a given layout might be based on the distance to the next advert in the same category and the distance to another advert of the same series. If you later have time of day considerations you can easily add them to the score function.
Initially you allocate the adverts sequentially, evenly or randomly within their time window (doesn't really matter which). Now you pick two slots and consider what happens to the score when you switch the contents of those two slots. If either advert moves out of its allowed range you can reject the change immediately. If both are still in range, does it move you to a better overall score? Initially you take changes randomly even if they make it worse but over time you reduce the probability of that happening so that by the end you are moving monotonically towards a better score.
Easy to implement, easy to add new 'rules' that affect score, can easily adjust run-time to accept a 'good enough' answer, ...
Another approach would be to use a genetic algorithm, see this similar question: Best Fit Scheduling Algorithm this is likely harder to program but will probably converge more quickly on a good answer.