SparkSQL - Error in Schema [duplicate] - apache-spark-sql

What does ArrayIndexOutOfBoundsException mean and how do I get rid of it?
Here is a code sample that triggers the exception:
String[] names = { "tom", "bob", "harry" };
for (int i = 0; i <= names.length; i++) {
System.out.println(names[i]);
}

Your first port of call should be the documentation which explains it reasonably clearly:
Thrown to indicate that an array has been accessed with an illegal index. The index is either negative or greater than or equal to the size of the array.
So for example:
int[] array = new int[5];
int boom = array[10]; // Throws the exception
As for how to avoid it... um, don't do that. Be careful with your array indexes.
One problem people sometimes run into is thinking that arrays are 1-indexed, e.g.
int[] array = new int[5];
// ... populate the array here ...
for (int index = 1; index <= array.length; index++)
{
System.out.println(array[index]);
}
That will miss out the first element (index 0) and throw an exception when index is 5. The valid indexes here are 0-4 inclusive. The correct, idiomatic for statement here would be:
for (int index = 0; index < array.length; index++)
(That's assuming you need the index, of course. If you can use the enhanced for loop instead, do so.)

if (index < 0 || index >= array.length) {
// Don't use this index. This is out of bounds (borders, limits, whatever).
} else {
// Yes, you can safely use this index. The index is present in the array.
Object element = array[index];
}
See also:
The Java Tutorials - Language Basics - Arrays
Update: as per your code snippet,
for (int i = 0; i<=name.length; i++) {
The index is inclusive the array's length. This is out of bounds. You need to replace <= by <.
for (int i = 0; i < name.length; i++) {

From this excellent article: ArrayIndexOutOfBoundsException in for loop
To put it briefly:
In the last iteration of
for (int i = 0; i <= name.length; i++) {
i will equal name.length which is an illegal index, since array indices are zero-based.
Your code should read
for (int i = 0; i < name.length; i++)
^

It means that you are trying to access an index of an array which is not valid as it is not in between the bounds.
For example this would initialize a primitive integer array with the upper bound 4.
int intArray[] = new int[5];
Programmers count from zero. So this for example would throw an ArrayIndexOutOfBoundsException as the upper bound is 4 and not 5.
intArray[5];

What causes ArrayIndexOutOfBoundsException?
If you think of a variable as a "box" where you can place a value, then an array is a series of boxes placed next to each other, where the number of boxes is a finite and explicit integer.
Creating an array like this:
final int[] myArray = new int[5]
creates a row of 5 boxes, each holding an int. Each of the boxes has an index, a position in the series of boxes. This index starts at 0 and ends at N-1, where N is the size of the array (the number of boxes).
To retrieve one of the values from this series of boxes, you can refer to it through its index, like this:
myArray[3]
Which will give you the value of the 4th box in the series (since the first box has an index of 0).
An ArrayIndexOutOfBoundsException is caused by trying to retrieve a "box" that does not exist, by passing an index that is higher than the index of the last "box", or negative.
With my running example, these code snippets would produce such an exception:
myArray[5] //tries to retrieve the 6th "box" when there is only 5
myArray[-1] //just makes no sense
myArray[1337] //way to high
How to avoid ArrayIndexOutOfBoundsException
In order to prevent ArrayIndexOutOfBoundsException, there are some key points to consider:
Looping
When looping through an array, always make sure that the index you are retrieving is strictly smaller than the length of the array (the number of boxes). For instance:
for (int i = 0; i < myArray.length; i++) {
Notice the <, never mix a = in there..
You might want to be tempted to do something like this:
for (int i = 1; i <= myArray.length; i++) {
final int someint = myArray[i - 1]
Just don't. Stick to the one above (if you need to use the index) and it will save you a lot of pain.
Where possible, use foreach:
for (int value : myArray) {
This way you won't have to think about indexes at all.
When looping, whatever you do, NEVER change the value of the loop iterator (here: i). The only place this should change value is to keep the loop going. Changing it otherwise is just risking an exception, and is in most cases not necessary.
Retrieval/update
When retrieving an arbitrary element of the array, always check that it is a valid index against the length of the array:
public Integer getArrayElement(final int index) {
if (index < 0 || index >= myArray.length) {
return null; //although I would much prefer an actual exception being thrown when this happens.
}
return myArray[index];
}

To avoid an array index out-of-bounds exception, one should use the enhanced-for statement where and when they can.
The primary motivation (and use case) is when you are iterating and you do not require any complicated iteration steps. You would not be able to use an enhanced-for to move backwards in an array or only iterate on every other element.
You're guaranteed not to run out of elements to iterate over when doing this, and your [corrected] example is easily converted over.
The code below:
String[] name = {"tom", "dick", "harry"};
for(int i = 0; i< name.length; i++) {
System.out.print(name[i] + "\n");
}
...is equivalent to this:
String[] name = {"tom", "dick", "harry"};
for(String firstName : name) {
System.out.println(firstName + "\n");
}

In your code you have accessed the elements from index 0 to the length of the string array. name.length gives the number of string objects in your array of string objects i.e. 3, but you can access only up to index 2 name[2],
because the array can be accessed from index 0 to name.length - 1 where you get name.length number of objects.
Even while using a for loop you have started with index zero and you should end with name.length - 1. In an array a[n] you can access form a[0] to a[n-1].
For example:
String[] a={"str1", "str2", "str3" ..., "strn"};
for(int i=0; i<a.length(); i++)
System.out.println(a[i]);
In your case:
String[] name = {"tom", "dick", "harry"};
for(int i = 0; i<=name.length; i++) {
System.out.print(name[i] +'\n');
}

For your given array the length of the array is 3(i.e. name.length = 3). But as it stores element starting from index 0, it has max index 2.
So, instead of 'i**<=name.length' you should write 'i<**name.length' to avoid 'ArrayIndexOutOfBoundsException'.

So much for this simple question, but I just wanted to highlight a new feature in Java which will avoid all confusions around indexing in arrays even for beginners. Java-8 has abstracted the task of iterating for you.
int[] array = new int[5];
//If you need just the items
Arrays.stream(array).forEach(item -> { println(item); });
//If you need the index as well
IntStream.range(0, array.length).forEach(index -> { println(array[index]); })
What's the benefit? Well, one thing is the readability like English. Second, you need not worry about the ArrayIndexOutOfBoundsException

The most common case I've seen for seemingly mysterious ArrayIndexOutOfBoundsExceptions, i.e. apparently not caused by your own array handling code, is the concurrent use of SimpleDateFormat. Particularly in a servlet or controller:
public class MyController {
SimpleDateFormat dateFormat = new SimpleDateFormat("MM/dd/yyyy");
public void handleRequest(ServletRequest req, ServletResponse res) {
Date date = dateFormat.parse(req.getParameter("date"));
}
}
If two threads enter the SimplateDateFormat.parse() method together you will likely see an ArrayIndexOutOfBoundsException. Note the synchronization section of the class javadoc for SimpleDateFormat.
Make sure there is no place in your code that are accessing thread unsafe classes like SimpleDateFormat in a concurrent manner like in a servlet or controller. Check all instance variables of your servlets and controllers for likely suspects.

You are getting ArrayIndexOutOfBoundsException due to i<=name.length part. name.length return the length of the string name, which is 3. Hence when you try to access name[3], it's illegal and throws an exception.
Resolved code:
String[] name = {"tom", "dick", "harry"};
for(int i = 0; i < name.length; i++) { //use < insteadof <=
System.out.print(name[i] +'\n');
}
It's defined in the Java language specification:
The public final field length, which contains the number of components
of the array. length may be positive or zero.

That's how this type of exception looks when thrown in Eclipse. The number in red signifies the index you tried to access. So the code would look like this:
myArray[5]
The error is thrown when you try to access an index which doesn't exist in that array. If an array has a length of 3,
int[] intArray = new int[3];
then the only valid indexes are:
intArray[0]
intArray[1]
intArray[2]
If an array has a length of 1,
int[] intArray = new int[1];
then the only valid index is:
intArray[0]
Any integer equal to the length of the array, or bigger than it: is out of bounds.
Any integer less than 0: is out of bounds;
P.S.: If you look to have a better understanding of arrays and do some practical exercises, there's a video here: tutorial on arrays in Java

For multidimensional arrays, it can be tricky to make sure you access the length property of the right dimension. Take the following code for example:
int [][][] a = new int [2][3][4];
for(int i = 0; i < a.length; i++){
for(int j = 0; j < a[i].length; j++){
for(int k = 0; k < a[j].length; k++){
System.out.print(a[i][j][k]);
}
System.out.println();
}
System.out.println();
}
Each dimension has a different length, so the subtle bug is that the middle and inner loops use the length property of the same dimension (because a[i].length is the same as a[j].length).
Instead, the inner loop should use a[i][j].length (or a[0][0].length, for simplicity).

For any array of length n, elements of the array will have an index from 0 to n-1.
If your program is trying to access any element (or memory) having array index greater than n-1, then Java will throw ArrayIndexOutOfBoundsException
So here are two solutions that we can use in a program
Maintaining count:
for(int count = 0; count < array.length; count++) {
System.out.println(array[count]);
}
Or some other looping statement like
int count = 0;
while(count < array.length) {
System.out.println(array[count]);
count++;
}
A better way go with a for each loop, in this method a programmer has no need to bother about the number of elements in the array.
for(String str : array) {
System.out.println(str);
}

ArrayIndexOutOfBoundsException whenever this exception is coming it mean you are trying to use an index of array which is out of its bounds or in lay man terms you are requesting more than than you have initialised.
To prevent this always make sure that you are not requesting a index which is not present in array i.e. if array length is 10 then your index must range between 0 to 9

ArrayIndexOutOfBounds means you are trying to index a position within an array that is not allocated.
In this case:
String[] name = { "tom", "dick", "harry" };
for (int i = 0; i <= name.length; i++) {
System.out.println(name[i]);
}
name.length is 3 since the array has been defined with 3 String objects.
When accessing the contents of an array, position starts from 0. Since there are 3 items, it would mean name[0]="tom", name[1]="dick" and name[2]="harry
When you loop, since i can be less than or equal to name.length, you are trying to access name[3] which is not available.
To get around this...
In your for loop, you can do i < name.length. This would prevent looping to name[3] and would instead stop at name[2]
for(int i = 0; i<name.length; i++)
Use a for each loop
String[] name = { "tom", "dick", "harry" };
for(String n : name) {
System.out.println(n);
}
Use list.forEach(Consumer action) (requires Java8)
String[] name = { "tom", "dick", "harry" };
Arrays.asList(name).forEach(System.out::println);
Convert array to stream - this is a good option if you want to perform additional 'operations' to your array e.g. filter, transform the text, convert to a map etc (requires Java8)
String[] name = { "tom", "dick", "harry" };
--- Arrays.asList(name).stream().forEach(System.out::println);
--- Stream.of(name).forEach(System.out::println);

ArrayIndexOutOfBoundsException means that you are trying to access an index of the array that does not exist or out of the bound of this array. Array indexes start from 0 and end at length - 1.
In your case
for(int i = 0; i<=name.length; i++) {
System.out.print(name[i] +'\n'); // i goes from 0 to length, Not correct
}
ArrayIndexOutOfBoundsException happens when you are trying to access
the name.length indexed element which does not exist (array index ends at length -1). just replacing <= with < would solve this problem.
for(int i = 0; i < name.length; i++) {
System.out.print(name[i] +'\n'); // i goes from 0 to length - 1, Correct
}

According to your Code :
String[] name = {"tom", "dick", "harry"};
for(int i = 0; i<=name.length; i++) {
System.out.print(name[i] +'\n');
}
If You check
System.out.print(name.length);
you will get 3;
that mean your name length is 3
your loop is running from 0 to 3
which should be running either "0 to 2" or "1 to 3"
Answer
String[] name = {"tom", "dick", "harry"};
for(int i = 0; i<name.length; i++) {
System.out.print(name[i] +'\n');
}

Each item in an array is called an element, and each element is accessed by its numerical index. As shown in the preceding illustration, numbering begins with 0. The 9th element, for example, would therefore be accessed at index 8.
IndexOutOfBoundsException is thrown to indicate that an index of some sort (such as to an array, to a string, or to a vector) is out of range.
Any array X, can be accessed from [0 to (X.length - 1)]

I see all the answers here explaining how to work with arrays and how to avoid the index out of bounds exceptions. I personally avoid arrays at all costs. I use the Collections classes, which avoids all the silliness of having to deal with array indices entirely. The looping constructs work beautifully with collections supporting code that is both easier to write, understand and maintain.

If you use an array's length to control iteration of a for loop, always remember that the index of the first item in an array is 0. So the index of the last element in an array is one less than the array's length.

ArrayIndexOutOfBoundsException name itself explains that If you trying to access the value at the index which is out of the scope of Array size then such kind of exception occur.
In your case, You can just remove equal sign from your for loop.
for(int i = 0; i<name.length; i++)
The better option is to iterate an array:
for(String i : name )
System.out.println(i);

This error is occurs at runs loop overlimit times.Let's consider simple example like this,
class demo{
public static void main(String a[]){
int[] numberArray={4,8,2,3,89,5};
int i;
for(i=0;i<numberArray.length;i++){
System.out.print(numberArray[i+1]+" ");
}
}
At first, I have initialized an array as 'numberArray'. then , some array elements are printed using for loop. When loop is running 'i' time , print the (numberArray[i+1] element..(when i value is 1, numberArray[i+1] element is printed.)..Suppose that, when i=(numberArray.length-2), last element of array is printed..When 'i' value goes to (numberArray.length-1) , no value for printing..In that point , 'ArrayIndexOutOfBoundsException' is occur.I hope to you could get idea.thank you !

You can use Optional in functional style to avoid NullPointerException and ArrayIndexOutOfBoundsException :
String[] array = new String[]{"aaa", null, "ccc"};
for (int i = 0; i < 4; i++) {
String result = Optional.ofNullable(array.length > i ? array[i] : null)
.map(x -> x.toUpperCase()) //some operation here
.orElse("NO_DATA");
System.out.println(result);
}
Output:
AAA
NO_DATA
CCC
NO_DATA

In most of the programming language indexes is start from 0.So you must have to write i<names.length or i<=names.length-1 instead of i<=names.length.

You could not iterate or store more data than the length of your array. In this case you could do like this:
for (int i = 0; i <= name.length - 1; i++) {
// ....
}
Or this:
for (int i = 0; i < name.length; i++) {
// ...
}

Related

Looping over an NSmutatable Array starting from a certain index

I have a quick question how can I loop over an NSMutable array starting from a certain index.
For Example I have these double loops I want k to start from the same index as l.
for (Line *l in L)
{
for (Line *k in L)
{
............
}
}
To elaborate further, lets say L has 10 object so l start from 0-10 and k from 0 -10. What I want is if l is equal 1 k should start from 1-10 rather than 0 - 10 and when l is equal 2 k should start from 2- 10 rather than 0. Any help is Appreciated
Objective-C is an extension of C, lookup the C for loop and you'll have your answer. HTH
Addendum
I was going to let you benefit from the learning experience of looking up the C for yourself, however at the time of writing all other answers since added give the code but it is not complete, so here is what you need to produce the l and k values in the order you wish:
for(NSInteger lIndex = 0; lIndex < L.count; lIndex++)
{
Line *l = L[lIndex]; // index into your array to get the element
for(NSInteger kIndex = lIndex; kIndex < L.count; kIndex++)
{
Line *k = L[kIndex];
// process your l and k
}
}
As you can see the for has three sub-parts which are the initialisation, condition, and increment. The initialisation is performed first, then the condition to determine whether to execute the for body, and the increment is executed after the statements in the body and before the condition is tested to determine if another iteration should be performed. A for loop is roughly (there are some differences that are unimportant here) to the while loop:
initialisation;
while(condition)
{
body statements;
increment;
}
You simply need to modify for-statement.
NSInteger indexYouNeed;
NSInteger iterationCount;
for (int i = indexYouNeed; i < iterationCount; i++) {
/* Your code here */
}
You may find this link helpfulll.
You have to use an indexed (ordinary) for loop instead of fast enumeration (for-in):
int l;
for (l=startValue; l<=endValue; l++)
{
int i;
for (int i=l; i<=endValue; i++)
{
…
}
}

Selection sort implementation, I am stuck at calculating time complexity for number of swaps

static int count = 0;
for (int i = 0; i < arr.length; i++) {
for (int j = i + 1; j < arr.length; j++) {
if (arr[i] > arr[j]) {
swap(arr, i, j);
count++;
}
}
}
Is this the correct implementation for selection sort? I am not getting O(n-1) complexity for swaps with this implementation.
Is this the correct implementation for selection sort?
It depends, logically what you are doing is correct. It sort using "find the max/min value in the array". But, in Selection Sort, usually you didn't need more than one swap in one iteration. You just save the max/min value in the array, then at the end you swap it with the i-th element
I am not getting O(n-1) complexity for swaps
did you mean n-1 times of swap? yes, it happen because you swap every times find a larger value not only on the largest value. You can try to rewrite your code like this:
static int count=0;
static int maximum=0;
for(int i=0;i<arr.length-1;i++){
maximum = i;
for(int j=i+1;j<arr.length;j++){
if(arr[j] > arr[maximum]){
maximum = j;
}
}
swap(arr[maximum],arr[i]);
count++;
}
Also, if you want to exact n-1 times swap, your iteration for i should changed too.

array lists in java: exclude the first element from `for` loop

I am taking an introduction to Java programing class and I have an array list where I need to exclude the first element from my for loop that finds an average. The first element in the array list is a weight for the average (which is why it needs to be excluded). I also need to drop the lowest value from the remainder of the array list hence my second for loop. I have tried to create a copy of the list and also tried to create a sub list but I cannot get it to work.
public static double Avgerage(ArrayList<Double> inputValues) {
double avg;
double sum = 0;
double weightValue = inputValues.get(0);
double lowest = inputValues.get(0);
for (int i = 1; i > inputValues.size(); i++) {
if (inputValues.get(i) < lowest) {
lowest = inputValues.get(i);
}
}
for (int i = 0; i < inputValues.size(); i++) {
sum = sum + inputValues.get(i);
}
double average = (sum - lowest) / (inputValues.size() - 1);
avg = average * weightValue;
return avg;
}
To start with good programming practice, you should work with interfaces rather than classes, where possible. The appropriate interface here is List<Double>, and when you create it in your class, you should use
List<Double> nameOfList = new ArrayList<Double>();
What we're doing is creating an object which has the behaviour of a List, with the underlying implementation of an ArrayList (more info here.
With regards to the question, you don't appear to be excluding the first element, as you said you wished to - both for loops iterate through all values in the list. Remember to treat the ArrayList like an array - accessing an element does not modify it, like it might in a Queue.
I have edited your code below to demonstrate this, and have also included some other optimisations and corrected the sign error on line 7:
public static double average(List<Double> inputValues) {
double sum = 0;
//Exclude the first element, as it contains the weight
double lowest = inputValues.get(1);
for (int i = 2; i < inputValues.size(); i++) {
lowest = Math.min(inputValues.get(i), lowest);
}
for (int i = 1; i < inputValues.size(); i++) {
sum += inputValues.get(i);
}
double average = (sum - lowest) / (inputValues.size() - 1);
//Scale by the weight
avg *= inputValues.get(0);
return avg;
}
Note: The convention in java is to use camelCase for method names, I have adjusted accordingly.
Also, I don't know your requirements, but optimally, you should be providing logical parameters. If possible do the following before calling the function:
int weight = inputValues.get(0);
inputValues.remove(0);
//And then you would call like this, and update your method signature to match
average(inputValues, weight);
I don't do this inside the method, as the context implies that we would not be modifying values.

What's the term for saving values of calculations instead of recalculating multiple times?

When you have code like this (written in java, but applicable to any similar language):
public static void main(String[] args) {
int total = 0;
for (int i = 0; i < 50; i++)
total += i * doStuff(i % 2); // multiplies i times doStuff(remainder of i / 2)
}
public static int doStuff(int i) {
// Lots of complicated calculations
}
You can see that there's room for improvement. doStuff(i % 2) only returns two different values - one for doStuff(0) on even numbers and one for doStuff(1) on odd numbers. Therefore you're wasting a lot of computation time/power on recalculating those values each time by saying doStuff(i % 2). You can improve like this:
public static void main(String[] args) {
int total = 0;
boolean[] alreadyCalculated = new boolean[2];
int[] results = new int[2];
for (int i = 0; i < 50; i++) {
if (!alreadyCalculated[i % 2]) {
results[i % 2] = doStuff(i % 2);
alreadyCalculated[i % 2] = true;
}
total += i * results[i % 2];
}
}
Now it accesses a stored value instead of recalculating each time. It might seem silly to keep arrays like that, but for cases like looping from, say, i = 0, i < 500 and you're checking i % 32 each time, or something, an array is an elegant approach.
Is there a term for this kind of code optimization? I'd like to read up more on the different forms and the conventions of it but I'm lacking a concise description.
Is there a term for this kind of code optimization?
Yes, there is:
In computing, memoization is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.
https://en.wikipedia.org/wiki/Memoization
Common-subexpression-elimination (CSE) is related to this. This case is a combination of that and hoisting a loop-invariant calculation out of a loop.
I'd agree with CBroe that you could call this specific form of caching memoization, esp the way you're implementing it with the clunky alreadyCalculated array. You can optimize that away since you know which calls will be new values and which will be repeats. Normally you'd implement memoization with a static array inside the called function, for the benefit of all callers. Ideally there's a sentinel value you can use to mark entries which don't have a result computed yet, instead of maintaining a separate array for that. Or for a sparse set of input values, just use a hash (instead of e.g. an array with 2^32 entries).
You can also avoid the if in the main loop.
public class Optim
{
public static int doStuff(int i) { return (i+5) << 1; }
public static void main(String[] args)
{
int total = 0;
int results[] = new int[2];
// more interesting if we pretend the loop count isn't known to be > 1, so avoiding calling doStuff(1) for n=1 is useful.
// otherwise you'd just do int[] results = { doStuff(0), doStuff(1) };
int n = 50;
for (int i = 0 ; i < Math.min(n, 2) ; i++) {
results[i] = doStuff(i);
total += i * results[i];
}
for (int i = 2; i < n; i++) { // runs zero times if n < 2
total += i * results[i % 2];
}
System.out.print(total);
}
}
Of course, in this case we can optimize a lot further. sum(0..n) = n * (n+1) / 2, so we can use that to get a closed-form (non-looping) solution in terms of doStuff(0) (sum of the even terms) and doStuff(1) (sum of the odd terms). So we only need the two doStuff() results once each, avoiding any need to memoize.

index out of bounds exception while creating file text [duplicate]

This question already has answers here:
What is an IndexOutOfRangeException / ArgumentOutOfRangeException and how do I fix it?
(5 answers)
Closed 7 years ago.
I'm getting one of the following errors:
"Index was out of range. Must be non-negative and less than the size of the collection"
"Insertion index was out of range. Must be non-negative and less than or equal to size."
"Index was outside the bounds of the array."
What does it mean, and how do I fix it?
See Also
IndexOutOfRangeException
ArgumentOutOfRangeException
Why does this error occur?
Because you tried to access an element in a collection, using a numeric index that exceeds the collection's boundaries.
The first element in a collection is generally located at index 0. The last element is at index n-1, where n is the Size of the collection (the number of elements it contains). If you attempt to use a negative number as an index, or a number that is larger than Size-1, you're going to get an error.
How indexing arrays works
When you declare an array like this:
var array = new int[6]
The first and last elements in the array are
var firstElement = array[0];
var lastElement = array[5];
So when you write:
var element = array[5];
you are retrieving the sixth element in the array, not the fifth one.
Typically, you would loop over an array like this:
for (int index = 0; index < array.Length; index++)
{
Console.WriteLine(array[index]);
}
This works, because the loop starts at zero, and ends at Length-1 because index is no longer less than Length.
This, however, will throw an exception:
for (int index = 0; index <= array.Length; index++)
{
Console.WriteLine(array[index]);
}
Notice the <= there? index will now be out of range in the last loop iteration, because the loop thinks that Length is a valid index, but it is not.
How other collections work
Lists work the same way, except that you generally use Count instead of Length. They still start at zero, and end at Count - 1.
for (int index = 0; i < list.Count; index++)
{
Console.WriteLine(list[index]);
}
However, you can also iterate through a list using foreach, avoiding the whole problem of indexing entirely:
foreach (var element in list)
{
Console.WriteLine(element.ToString());
}
You cannot index an element that hasn't been added to a collection yet.
var list = new List<string>();
list.Add("Zero");
list.Add("One");
list.Add("Two");
Console.WriteLine(list[3]); // Throws exception.