(I am new to pyspark)
I am trying to read a csv file into a pyspark dataframe as follows:
from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession, SQLContext
spark = SparkSession.builder.master("local[1]").appName("SampleWork").getOrCreate()
df = spark.read.csv('train.csv',sep=",",header=True,inferSchema=True)
But I am not getting expected result dataframe here. How to read this file correctly.
I have added the first 3 rows of the sample csv file here.
csv file:
id,url_legal,license,excerpt,target,standard_error
c12129c31,,,"When the young people returned to the ballroom, it presented a decidedly changed appearance. Instead of an interior scene, it was a winter landscape.
The floor was covered with snow-white canvas, not laid on smoothly, but rumpled over bumps and hillocks, like a real snow field. The numerous palms and evergreens that had decorated the room, were powdered with flour and strewn with tufts of cotton, like snow. Also diamond dust had been lightly sprinkled on them, and glittering crystal icicles hung from the branches.
At each end of the room, on the wall, hung a beautiful bear-skin rug.
These rugs were for prizes, one for the girls and one for the boys. And this was the game.
The girls were gathered at one end of the room and the boys at the other, and one end was called the North Pole, and the other the South Pole. Each player was given a small flag which they were to plant on reaching the Pole.
This would have been an easy matter, but each traveller was obliged to wear snowshoes.",-0.340259125,0.464009046
85aa80a4c,,,"All through dinner time, Mrs. Fayre was somewhat silent, her eyes resting on Dolly with a wistful, uncertain expression. She wanted to give the child the pleasure she craved, but she had hard work to bring herself to the point of overcoming her own objections.
At last, however, when the meal was nearly over, she smiled at her little daughter, and said, ""All right, Dolly, you may go.""
""Oh, mother!"" Dolly cried, overwhelmed with sudden delight. ""Really?
Oh, I am so glad! Are you sure you're willing?""
""I've persuaded myself to be willing, against my will,"" returned Mrs. Fayre, whimsically. ""I confess I just hate to have you go, but I can't bear to deprive you of the pleasure trip. And, as you say, it would also keep Dotty at home, and so, altogether, I think I shall have to give in.""
""Oh, you angel mother! You blessed lady! How good you are!"" And Dolly flew around the table and gave her mother a hug that nearly suffocated her.",-0.315372342,0.480804970
b69ac6792,,,"As Roger had predicted, the snow departed as quickly as it came, and two days after their sleigh ride there was scarcely a vestige of white on the ground. Tennis was again possible and a great game was in progress on the court at Pine Laurel. Patty and Roger were playing against Elise and Sam Blaney, and the pairs were well matched.
But the long-contested victory finally went against Patty, and she laughingly accepted defeat.
""Only because Patty's not quite back on her game yet,"" Roger defended; ""this child has been on the sick list, you know, Sam, and she isn't up to her own mark.""
""Well, I like that!"" cried Patty; ""suppose you bear half the blame, Roger. You see, Mr. Blaney, he is so absorbed in his own Love Game, he can't play with his old-time skill.""
""All right, Patsy, let it go at that. And it's so, too. I suddenly remembered something Mona told me to tell you, and it affected my service.""",-0.580117966,0.476676226
You can use the custom escape character with the multiLine option.
df = spark.read.csv("test.csv", header=True, inferSchema=True, escape="\"", multiLine=True)
+---------+---------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+
|id |url_legal|license|excerpt |target |standard_error|
+---------+---------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+
|c12129c31|null |null |When the young people returned to the ballroom, it presented a decidedly changed appearance. Instead of an interior scene, it was a winter landscape.
The floor was covered with snow-white canvas, not laid on smoothly, but rumpled over bumps and hillocks, like a real snow field. The numerous palms and evergreens that had decorated the room, were powdered with flour and strewn with tufts of cotton, like snow. Also diamond dust had been lightly sprinkled on them, and glittering crystal icicles hung from the branches.
At each end of the room, on the wall, hung a beautiful bear-skin rug.
These rugs were for prizes, one for the girls and one for the boys. And this was the game.
The girls were gathered at one end of the room and the boys at the other, and one end was called the North Pole, and the other the South Pole. Each player was given a small flag which they were to plant on reaching the Pole.
This would have been an easy matter, but each traveller was obliged to wear snowshoes.|-0.340259125|0.464009046 |
|85aa80a4c|null |null |All through dinner time, Mrs. Fayre was somewhat silent, her eyes resting on Dolly with a wistful, uncertain expression. She wanted to give the child the pleasure she craved, but she had hard work to bring herself to the point of overcoming her own objections.
At last, however, when the meal was nearly over, she smiled at her little daughter, and said, "All right, Dolly, you may go."
"Oh, mother!" Dolly cried, overwhelmed with sudden delight. "Really?
Oh, I am so glad! Are you sure you're willing?"
"I've persuaded myself to be willing, against my will," returned Mrs. Fayre, whimsically. "I confess I just hate to have you go, but I can't bear to deprive you of the pleasure trip. And, as you say, it would also keep Dotty at home, and so, altogether, I think I shall have to give in."
"Oh, you angel mother! You blessed lady! How good you are!" And Dolly flew around the table and gave her mother a hug that nearly suffocated her. |-0.315372342|0.48080497 |
|b69ac6792|null |null |As Roger had predicted, the snow departed as quickly as it came, and two days after their sleigh ride there was scarcely a vestige of white on the ground. Tennis was again possible and a great game was in progress on the court at Pine Laurel. Patty and Roger were playing against Elise and Sam Blaney, and the pairs were well matched.
But the long-contested victory finally went against Patty, and she laughingly accepted defeat.
"Only because Patty's not quite back on her game yet," Roger defended; "this child has been on the sick list, you know, Sam, and she isn't up to her own mark."
"Well, I like that!" cried Patty; "suppose you bear half the blame, Roger. You see, Mr. Blaney, he is so absorbed in his own Love Game, he can't play with his old-time skill."
"All right, Patsy, let it go at that. And it's so, too. I suddenly remembered something Mona told me to tell you, and it affected my service." |-0.580117966|0.476676226 |
+---------+---------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+
Related
I made a program that shows a large amount of text in a rich text box in the latest version of Visual Studio on my home computer, and even though there are line breaks in the text, the program picks up everything that is within the quotes (eg. TextBox1.Text = "multiple paragraphs of writing", and everything within those quotes would turn red like output text is supposed to. Now, when I put the program on my school computer running Visual Studio 2013, it's only picking up the first line of text within the quotes and ignoring the other lines, and thus can not run because all of those lines of text are causing errors. Why is this happening?
Private Sub SnowyOwl_Load(sender As Object, e As EventArgs) Handles MyBase.Load
SNOtext.Text = "The regal Snowy Owl is one of the few birds that can get even non-birders to come out for a look. This largest (by weight) North American owl shows up irregularly in winter to hunt in windswept fields or dunes, a pale shape with catlike yellow eyes. They spend summers far north of the Arctic Circle hunting lemmings, ptarmigan, and other prey in 24-hour daylight. In years of lemming population booms they can raise double or triple the usual number of young.
MEASUREMENTS:
Length:
20.5–28 in
Wingspan:
49.6–57.1 in
Weight:
56.4–104.1 oz
SIZE AND SHAPE:
- Snowy Owls are very large owls with smoothly rounded heads and no ear tufts, and are about the same size as a Great Horned Owl. The body is bulky, with dense feathering on the legs that makes the bird look wide at the base when sitting on the ground.
COLOR PATTERN:
- Snowy Owls are white birds with varying amounts of black or brown markings on the body and wings. On females this can be quite dense, giving the bird a salt-and-pepper look. Males tend to be paler and become whiter as they age. The eyes are yellow.
BEHAVIOR:
- Look for Snowy Owls sitting on or near the ground in wide-open areas. They often perch on rises such as the crests of dunes, or on fenceposts, telephone poles, and hay bales. When they fly they usually stay close to the ground.
MIGRATION:
- Irruptive medium- to long-distance migrant. Snowy Owl migrations are extremely variable. Some North American Snowy Owls winter in southeastern Canada, the upper Great Lakes states, and New England just about every year. Winter numbers of Snowy Owls in the U.S. peak periodically, which may be attributed to lemming cycles farther north. During irruptive years, Snowy Owls can flush south throughout the lower 48 states, as far as south as Texas and Florida in extreme years.
FIND THIS BIRD:
- Unless you visit the high arctic, you’ll mainly be looking for Snowy Owls during winter in wide-open areas such as fields and shorelines. Scan snowy flat areas and be on the lookout for any irregularities in the snow. A lump or dirty patch could be a Snowy Owl facing away from you. Snowy Owls like to perch in conspicuous areas, so be sure to check high points like hay bales, fenceposts, telephone poles, buildings, or grain elevators. Also look for agitated birds—other raptors or gulls may swoop at a Snowy Owl sitting on a beach.
HABITAT:
- In winter, look for Snowy Owls along shorelines of lakes and the ocean, as well as on agricultural fields and airport lands. Snowy Owls breed in the treeless arctic tundra.
FUN FACTS:
- The Snowy Owl can be found represented in cave paintings in Europe.
- In some years, some North American Snowy Owls remain on their breeding grounds year-round, while others migrate in winter to southern Canada and the northern half of the contiguous United States. In the northern plains, New York, and New England, Snowy Owls occur regularly in winter. Elsewhere, such as in the Pacific Northwest, the Midwest, and eastern Canada, Snowy Owls are irruptive, appearing only in some winters but not in others.
- Male Snowy Owls are barred with dark brown when they’re young and get whiter as they get older. Females keep some dark markings throughout their lives. Although the darkest males and the palest females are nearly alike in color, the whitest birds—including the ones that played Harry Potter’s Hedwig—are always males and the most heavily barred ones are always females.
- Unlike most owls, Snowy Owls are diurnal, extremely so. They’ll hunt at all hours during the continuous daylight of an Arctic summer. And they may eat more than 1,600 lemmings, their favorite meal, in a single year.
- Snowy Owl young may disperse remarkably far from their birthplace. From a single Snowy Owl nest on Victoria Island in the Canadian Arctic, one young bird went to Hudson Bay, one to southeastern Ontario, and one to the far eastern Russian coast.
- John James Audubon once saw a Snowy Owl lying at the edge of an ice hole, where it waited for fish and caught them using its feet.
-The oldest-known Snowy Owl was a female, and at least 23 years, 10 months old when she was recaptured in 2015 during banding operations in Montana. She had been banded in Massachusetts in 1992.
All readings provided above are courtesy of the Cornell Lab of Ornithology and AllAboutBirds.org."
End Sub
OK I have read a lot of things on the web most of them using Java that doesn't have unsigned int. I am working on Objective-C that has unsigned int.
Lets consider the following scenario.
The board is like this:
A1 is on the lower left and is the least significant bit and H8 is on the top right and is the most significant bit.
I have constructed the bitmasks for the movements of all pieces for all positions on the board.
Suppose the following simple situation:
the game will start. All pieces are on their initial positions.
the user tries to move the knight on G1 to E2.
Obviously this movement is impossible because there is a pawn on E2.
How do I discover if a movement is valid?
Ok I have the bitmask of all pieces that is this:
1111111111111111000000000000000000000000000000001111111111111111
and I have the bitmask of the possible movements for the knight at G1 that is
00000000
00000000
00000000
00000000
00000000
00000X0X
0000X000
00000000
or in bits
0000000000001000000001010000000000000000000000000000000000000000
if I AND these two
1111111111111111000000000000000000000000000000001111111111111111
0000000000001000000001010000000000000000000000000000000000000000
I get
0000000000001000000000000000000000000000000000000000000000000000
all I know is that there is a piece at that point but I am not seeing how this tells me if that is a valid move or not.
Another question is this: suppose this move would let my king under attack. Obviously the move would be technically possible under the knight rules of movement but impossible because I cannot do a move that puts my king under attack.
How do I get these informations? Thanks
If you only have a bitmask of all pieces then you simply do not have any way of knowing whether you're trying to move the knight onto an opponent's piece or your own. You will have to use a separate bitmask for white pieces and for the black pieces.
You already have the bitmask K of all possible moves of the knight before considering the locations of other pieces. Just bitwise AND this with the NOT of the where the white pieces are: M = K && !W
Now as far as I can think, the only way that a move of the knight would endanger its king is if the knight was previously blocking an opponent's attack. Specifically, only if it was blocking the attack of a rook or a bishop. This is only possible if the knight was positioned in one of the eight cardinal directions from the location of the king. So check to see if there is a straight, unblocked line from the king to knight in one of the eight cardinal directions. If there isn't, then the move could not possibly endanger the king, so you needn't do any more checks. If there is, then loop through the positions of M to check to see if the king will be endangered after each of them.
In the special case of the knight, since it travels in an L shape, there is no way that it can move from a position where it was blocking this attack to another position where it continued to thwart it. So in the special case of the knight, you should do the following:
Check if there is an uninterrupted straight line from the king to the knight. If not, go to 4.
Check if there is an opponent rook or bishop along this line whose only obstacle to taking the king is the knight you are trying to move. If not, go to 4.
If you've reached this point, then moving the knight would result in check. It cannot be moved. Set M = 0 and continue with the program.
Set M = K && !W and continue with the program.
I should note that this was written assuming that the king did not begin in check. Adjust as necessary.
i have the following paragraph:
Free Daily Horoscope by findyourfate.comhttp://www.findyourfate.comThings are good and getting better -- and you need to learn how to accept this fact. While this may sound like flippant or sarcastic advice, the truth is that lately, whenever the fates have thrown a few rose petals in your path, you have chosen to see them as just more debris -- if you have elected to see them at all. Wake up and wise up. You need to start acknowledging the gifts as gifts meant for you. You need to understand just how worthy of them you are.http://www.findyourfate.com/rss/dailyhoroscope-feed.asp?sign=Aries [return delivery report]
which I stored in a string using vb.net.
How can I split that string to output only the following part:
Things are good and getting better -- and you need to learn how to accept this fact. While this may sound like flippant or sarcastic advice, the truth is that lately, whenever the fates have thrown a few rose petals in your path, you have chosen to see them as just more debris -- if you have elected to see them at all. Wake up and wise up. You need to start acknowledging the gifts as gifts meant for you. You need to understand just how worthy of them you are.
In case you have this same paragraph and each time you need to get same output you can use replace instead of split it's easier:
Dim texts As String = "Free Daily Horoscope by findyourfate.comhttp://www.findyourfate.comThings are good and getting better -- and you need to learn how to accept this fact. While this may sound like flippant or sarcastic advice, the truth is that lately, whenever the fates have thrown a few rose petals in your path, you have chosen to see them as just more debris -- if you have elected to see them at all. Wake up and wise up. You need to start acknowledging the gifts as gifts meant for you. You need to understand just how worthy of them you are.http://www.findyourfate.com/rss/dailyhoroscope-feed.asp?sign=Aries [return delivery report]"
texts = texts.Replace("Free Daily Horoscope by findyourfate.comhttp://www.findyourfate.com", "")
texts = texts.Replace("http://www.findyourfate.com/rss/dailyhoroscope-feed.asp?sign=Aries [return delivery report]", "")
and you will get the same outcome that's of course if you have always
If you wish to use split:
Dim arraytext() As String
arraytext = Split(texts, "Free Daily Horoscope by findyourfate.comhttp://www.findyourfate.com")
arraytext = Split(arraytext(1), "http://www.findyourfate.com/rss/dailyhoroscope-feed.asp?sign=Aries [return delivery report]")
MsgBox(arraytext(0))
I have a game I am working on that has homing missiles in it. At the moment they just turn towards their target, which produces a rather dumb looking result, with all the missiles following the target around.
I want to create a more deadly flavour of missile that will aim at the where the target "will be" by the time it gets there and I am getting a bit stuck and confused about how to do it.
I am guessing I will need to work out where my target will be at some point in the future (a guess anyway), but I can't get my head around how far ahead to look. It needs to be based on how far the missile is away from the target, but the target it also moving.
My missiles have a constant thrust, combined with a weak ability to turn. The hope is they will be fast and exciting, but steer like a cow (ie, badly, for the non HitchHiker fans out there).
Anyway, seemed like a kind of fun problem for Stack Overflow to help me solve, so any ideas, or suggestions on better or "more fun" missiles would all be gratefully received.
Next up will be AI for dodging them ...
What you are suggesting is called "Command Guidance" but there is an easier, and better way.
The way that real missiles generally do it (Not all are alike) is using a system called Proportional Navigation. This means the missile "turns" in the same direction as the line-of-sight (LOS) between the missile and the target is turning, at a turn rate "proportional" to the LOS rate... This will do what you are asking for as when the LOS rate is zero, you are on collision course.
You can calculate the LOS rate by just comparing the slopes of the line between misile and target from one second to the next. If that slope is not changing, you are on collision course. if it is changing, calculate the change and turn the missile by a proportionate angular rate... you can use any metrics that represent missile and target position.
For example, if you use a proportionality constant of 2, and the LOS is moving to the right at 2 deg/sec, turn the missile to the right at 4 deg/sec. LOS to the left at 6 deg/sec, missile to the left at 12 deg/sec...
In 3-d problem is identical except the "Change in LOS Rate", (and resultant missile turn rate) is itself a vector, i.e., it has not only a magnitude, but a direction (Do I turn the missile left, right or up or down or 30 deg above horizontal to the right, etc??... Imagine, as a missile pilot, where you would "set the wings" to apply the lift...
Radar guided missiles, which "know" the rate of closure. adjust the proportionality constant based on closure (the higher the closure the faster the missile attempts to turn), so that the missile will turn more aggressively in high closure scenarios, (when the time of flight is lower), and less aggressively in low closure (tail chases) when it needs to conserve energy.
Other missiles (like Sidewinders), which do not know the closure, use a constant pre-determined proportionality value). FWIW, Vietnam era AIM-9 sidewinders used a proportionality constant of 4.
I've used this CodeProject article before - it has some really nice animations to explain the math.
"The Mathematics of Targeting and Simulating a Missile: From Calculus to the Quartic Formula":
http://www.codeproject.com/KB/recipes/Missile_Guidance_System.aspx
(also, hidden in the comments at the bottom of that article is a reference to some C++ code that accomplishes the same task from the Unreal wiki)
Take a look at OpenSteer. It has code to solve problems like this. Look at 'steerForSeek' or 'steerForPursuit'.
Have you considered negative feedback on the recent change of bearing over change of time?
Details left as an exercise.
The suggestions is completely serious: if the target does not maneuver this should obtain a near optimal intercept. And it should converge even if the target is actively dodging.
Need more detail?
Solving in a two dimensional space for ease of notation. Take \vec{m} to be the location of the missile and vector \vec{t} To be the location of the target.
The current heading in the direction of motion over last time unit: \vec{h} = \bar{\vec{m}_i - \vec{m}_i-1}}. Let r be the normlized vector between the missile and the target: \vec{r} = \bar{\vec{t} - \vec{m}}. The bearing is b = \vec{r} \dot \vec{h} Compute the bearing at each time tick, and the change thereof, and change heading to minimize that quantity.
The math is harrier in 3d because of the need to find the plane of action at each step, but the process is the same.
You'll want to interpolate the trajectory of both the target and the missile as a function of time. Then look for the times in which the coordinates of the objects are within some acceptable error.
I am would like to determine a direction moving x degrees clockwise starting on true north. Is there a way for me get or calculate true north based on a set of lat & long coordinates?
I am interested in implementing this cocoa touch. I am sure this is used in many of the applications already out there. Any comments, pointers, advice will be highly appreciated.
Thanks
ldj
if you get one set of coordinates, then another set say ten seconds later, you could work out which direction you had travelled in. (Not sure if this is what you wanted)
You would have to work out the change in longitude and latitude, then use a bit of trigonometry, and simple do arctan(long/lat) (arc tan is the inverse function of tan)
You'll have to avoid dividing by 0 when change in lat is 0. However, when the change in long or lat is 0 you know you have travelled directly north, east, south or west.
Also, arctan in most APIs outputs in radians so you must times it by 180 and divide by pi to get degrees.
Edit:
Is true north not located at any latitude of 0? Although my first impression this was grid north, I think due to the curve of the earth and the fact lines of longitude are drawn parallel creates variation referred to as grid north.
Do you mean magnetic north?
If so google for magnetic variation and you will have it.
Zekian is on the right path, though... It looks like you need your current long/lat, and the current magnetic poles long/lat, a bit of spherical geometry to figure out the magnetic derivation ("declination" is a word that pops up a lot), and then apply that derivation to your current magnetic north reading. And because the earth's magnetic field keeps moving around, you need to compensate for this too. Or atleast offer an update for the magnetic north's long/lat.
These are links I found along the way, but my head started spinning like in the Exorcist before I could come up with anything clever
http://www.nauticalissues.com/en/math1.html
Link
Link
http://www.ngdc.noaa.gov/geomag/magfield.shtml
http://en.wikipedia.org/wiki/Magnetic_declination