Get CSS property via Appium - selenium

I want to click on a button only if it is enabled "Green color". To find the color of a button I tried the below ways, but none worked and giving "UnsupportedCommandException"
MobileElement mobileElement = appiumDriver.findElement(By.xpath("//XCUIElementTypeButton[#name='Get Quote']"));
String color = mobileElement.getCssProperty("style");
String color = mobileElement.getCssProperty("background-color");
String color = mobileElement.getCssProperty("color");
Exception:
org.openqa.selenium.UnsupportedCommandException: Unhandled endpoint: {
wildcards = (
"session/element/EF010000-0000-0000-8007-000000000000/css/background-color"
);
} (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
Driver info: io.appium.java_client.ios.IOSDriver
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
at io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:239)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
at io.appium.java_client.DefaultGenericMobileDriver.execute(DefaultGenericMobileDriver.java:41)
at io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:1)
at io.appium.java_client.ios.IOSDriver.execute(IOSDriver.java:1)
at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:285)
at io.appium.java_client.DefaultGenericMobileElement.execute(DefaultGenericMobileElement.java:45)
at io.appium.java_client.MobileElement.execute(MobileElement.java:1)
at io.appium.java_client.ios.IOSElement.execute(IOSElement.java:1)
at org.openqa.selenium.remote.RemoteWebElement.getCssValue(RemoteWebElement.java:171)
at io.appium.java_client.DefaultGenericMobileElement.getCssValue(DefaultGenericMobileElement.java:177)
at io.appium.java_client.MobileElement.getCssValue(MobileElement.java:1)
at io.appium.java_client.ios.IOSElement.getCssValue(IOSElement.java:1)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at io.appium.java_client.pagefactory.ElementInterceptor.getObject(ElementInterceptor.java:40)
at io.appium.java_client.pagefactory.interceptors.InterceptorOfASingleElement.intercept(InterceptorOfASingleElement.java:60)
at io.appium.java_client.ios.IOSElement$$EnhancerByCGLIB$$d311658.getCssValue(<generated>)
at com.haulhub.pageactions.mobileapp.HomePageAction.clickPendingFilterButton(DemoPageAction.java:73)
at com.haulhub.tests.DotSlipDisplayFiltersTest.verifyPendingTicketMatchesPendingInspectorReview(DemoFiltersTest.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
at org.testng.SuiteRunner.run(SuiteRunner.java:240)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1198)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1123)
at org.testng.TestNG.run(TestNG.java:1031)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)

According to Appium documentation:
Get Element CSS Value command only applies to webview contexts
https://appium.io/docs/en/commands/element/attributes/css-property/
So it will not work for native context.
Check if element is enabled
There is another way to check if the button is enabled or not:
https://appium.io/docs/en/commands/element/attributes/enabled
MobileElement mobileElement = appiumDriver.findElement(By.xpath("//XCUIElementTypeButton[#name='Get Quote']"));
if(mobileElement.isEnabled()) {
mobileElement.click();
}
Check element color
If you also need to verify the element color, you should take the page (or element) screenshot, then get the element location and try to get the pixel color witing the element area on the screenshot by coordinates.
MobileElement mobileElement = appiumDriver.findElement(By.xpath("//XCUIElementTypeButton[#name='Get Quote']"));
File screenshot = ((TakesScreenshot)appiumDriver).getScreenshotAs(OutputType.FILE);
BufferedImage image = ImageIO.read(screenshot);
Point elementLocation = mobileElement.getLocation();
// Lets take color at point x=10, y=10 inside the element (0, 0 is the top left)
int checkColorX = elementLocation.getX() + 10;
int checkColorY = elementLocation.getY() + 10;
// Optionaly you may try to take not the page screenshot, but just the element screenshot
// File screenshot = mobileElement.getScreenshotAs(OutputType.FILE);
// BufferedImage image = ImageIO.read(screenshot);
// int checkColorX = 10;
// int checkColorY = 10;
// Getting pixel color by position x and y
int color = image.getRGB(checkColorX, checkColorY);
int red = (color & 0x00ff0000) >> 16;
int green = (color & 0x0000ff00) >> 8;
int blue = color & 0x000000ff;
System.out.println("Red Color value = "+ red);
System.out.println("Green Color value = "+ green);
System.out.println("Blue Color value = "+ blue);
// Then you need to compare each color with the values, you expect.

Related

JavaFX TableView: Exception in thread "JavaFX Application Thread" java.lang.IndexOutOfBoundsException

I am attempting to populate a JavaFx TableView with some data from a MySQL query run through JDBC. I previously had no problem with populating the table, until I changed the query to select all columns in the table except one of them. The query runs perfectly in MySQL Workbench.
This is the portion of my code responsible for populating the TableView:
ObservableList<ObservableList<String>> data = FXCollections.observableArrayList();
TableView table = new TableView();
// rs is the ResultSet returned by executing the SELECT query that should be filling the table
ResultSetMetaData rsmd = rs.getMetaData();
for(int i = 1; i <= rsmd.getColumnCount(); i++){
final int j = i;
TableColumn col = new TableColumn(rsmd.getColumnLabel(i));
col.setCellValueFactory(new Callback<CellDataFeatures<ObservableList,String>, ObservableValue<String>>(){
public ObservableValue<String> call(CellDataFeatures<ObservableList, String> param) {
// The line below is Main.java:194, the problem source according to the stack trace
return new SimpleStringProperty(param.getValue().get(j).toString());
}
});
table.getColumns().addAll(col);
}
while(rs.next()){
ObservableList<String> row = FXCollections.observableArrayList();
for(int i = 1 ; i <= rsmd.getColumnCount(); i++){
row.add(rs.getString(i));
}
data.add(row);
}
table.setItems(data);`
The stack trace indicates that the problem lies at the line indicated in the snippet. And here is the stack trace:
Exception in thread "JavaFX Application Thread"
java.lang.IndexOutOfBoundsException: Index: 25, Size: 25
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at com.sun.javafx.collections.ObservableListWrapper.get(ObservableListWrapper.java:89)
at socparser.Main$2.call(Main.java:194)
at socparser.Main$2.call(Main.java:1)
at javafx.scene.control.TableColumn.getCellObservableValue(TableColumn.java:578)
at javafx.scene.control.TableColumn.getCellObservableValue(TableColumn.java:563)
at javafx.scene.control.TableCell.updateItem(TableCell.java:644)
at javafx.scene.control.TableCell.indexChanged(TableCell.java:468)
at javafx.scene.control.IndexedCell.updateIndex(IndexedCell.java:116)
at com.sun.javafx.scene.control.skin.TableViewSkin.resizeColumnToFitContent(TableViewSkin.java:241)
at com.sun.javafx.scene.control.skin.TableViewSkin.resizeColumnToFitContent(TableViewSkin.java:54)
at com.sun.javafx.scene.control.skin.TableColumnHeader.doColumnAutoSize(TableColumnHeader.java:531)
at com.sun.javafx.scene.control.skin.TableColumnHeader.updateScene(TableColumnHeader.java:474)
at com.sun.javafx.scene.control.skin.TableColumnHeader.handlePropertyChanged(TableColumnHeader.java:314)
at com.sun.javafx.scene.control.skin.TableColumnHeader.lambda$new$49(TableColumnHeader.java:149)
at com.sun.javafx.scene.control.MultiplePropertyChangeListenerHandler$1.changed(MultiplePropertyChangeListenerHandler.java:55)
at javafx.beans.value.WeakChangeListener.changed(WeakChangeListener.java:89)
at com.sun.javafx.binding.ExpressionHelper$SingleChange.fireValueChangedEvent(ExpressionHelper.java:182)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.property.ReadOnlyObjectPropertyBase.fireValueChangedEvent(ReadOnlyObjectPropertyBase.java:74)
at javafx.beans.property.ReadOnlyObjectWrapper.fireValueChangedEvent(ReadOnlyObjectWrapper.java:102)
at javafx.scene.Node$ReadOnlyObjectWrapperManualFire.fireSuperValueChangedEvent(Node.java:831)
at javafx.scene.Node.invalidatedScenes(Node.java:881)
at javafx.scene.Node.setScenes(Node.java:919)
at javafx.scene.Parent$1.onChanged(Parent.java:269)
at com.sun.javafx.collections.TrackableObservableList.lambda$new$29(TrackableObservableList.java:45)
at com.sun.javafx.collections.ListListenerHelper$Generic.fireValueChangedEvent(ListListenerHelper.java:329)
at com.sun.javafx.collections.ListListenerHelper.fireValueChangedEvent(ListListenerHelper.java:73)
at javafx.collections.ObservableListBase.fireChange(ObservableListBase.java:233)
at javafx.collections.ListChangeBuilder.commit(ListChangeBuilder.java:482)
at javafx.collections.ListChangeBuilder.endChange(ListChangeBuilder.java:541)
at javafx.collections.ObservableListBase.endChange(ObservableListBase.java:205)
at javafx.collections.ModifiableObservableListBase.setAll(ModifiableObservableListBase.java:90)
at com.sun.javafx.collections.VetoableListDecorator.setAll(VetoableListDecorator.java:116)
at com.sun.javafx.scene.control.skin.NestedTableColumnHeader.updateContent(NestedTableColumnHeader.java:487)
at com.sun.javafx.scene.control.skin.NestedTableColumnHeader.updateTableColumnHeaders(NestedTableColumnHeader.java:317)
at com.sun.javafx.scene.control.skin.NestedTableColumnHeader.checkState(NestedTableColumnHeader.java:544)
at com.sun.javafx.scene.control.skin.NestedTableColumnHeader.computePrefHeight(NestedTableColumnHeader.java:427)
at javafx.scene.Parent.prefHeight(Parent.java:929)
at javafx.scene.layout.Region.prefHeight(Region.java:1435)
at com.sun.javafx.scene.control.skin.TableHeaderRow.computePrefHeight(TableHeaderRow.java:331)
at com.sun.javafx.scene.control.skin.TableHeaderRow.computeMinHeight(TableHeaderRow.java:324)
at javafx.scene.Parent.minHeight(Parent.java:957)
at javafx.scene.layout.Region.minHeight(Region.java:1401)
at javafx.scene.control.SkinBase.computeMinHeight(SkinBase.java:254)
at javafx.scene.control.Control.computeMinHeight(Control.java:489)
at javafx.scene.Parent.minHeight(Parent.java:957)
at javafx.scene.layout.Region.minHeight(Region.java:1401)
at javafx.scene.layout.Region.computeChildPrefAreaHeight(Region.java:1762)
at javafx.scene.layout.GridPane.computePrefHeights(GridPane.java:1424)
at javafx.scene.layout.GridPane.layoutChildren(GridPane.java:1690)
at javafx.scene.Parent.layout(Parent.java:1087)
at javafx.scene.Scene.doLayoutPass(Scene.java:552)
at javafx.scene.Scene$ScenePulseListener.pulse(Scene.java:2397)
at com.sun.javafx.tk.Toolkit.lambda$runPulse$29(Toolkit.java:398)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.tk.Toolkit.runPulse(Toolkit.java:397)
at com.sun.javafx.tk.Toolkit.firePulse(Toolkit.java:424)
at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:518)
at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:498)
at com.sun.javafx.tk.quantum.QuantumToolkit.pulseFromQueue(QuantumToolkit.java:491)
at com.sun.javafx.tk.quantum.QuantumToolkit.lambda$runToolkit$403(QuantumToolkit.java:319)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$147(WinApplication.java:177)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "JavaFX Application Thread"
java.lang.IndexOutOfBoundsException: Index: 25, Size: 25
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at com.sun.javafx.collections.ObservableListWrapper.get(ObservableListWrapper.java:89)
at socparser.Main$2.call(Main.java:194)
at socparser.Main$2.call(Main.java:1)
at javafx.scene.control.TableColumn.getCellObservableValue(TableColumn.java:578)
at javafx.scene.control.TableColumn.getCellObservableValue(TableColumn.java:563)
at javafx.scene.control.TableCell.updateItem(TableCell.java:644)
at javafx.scene.control.TableCell.indexChanged(TableCell.java:468)
at javafx.scene.control.IndexedCell.updateIndex(IndexedCell.java:116)
at com.sun.javafx.scene.control.skin.TableRowSkinBase.updateCells(TableRowSkinBase.java:533)
at com.sun.javafx.scene.control.skin.TableRowSkinBase.init(TableRowSkinBase.java:147)
at com.sun.javafx.scene.control.skin.TableRowSkin.<init>(TableRowSkin.java:64)
at javafx.scene.control.TableRow.createDefaultSkin(TableRow.java:212)
at javafx.scene.control.Control.impl_processCSS(Control.java:872)
at javafx.scene.Node.processCSS(Node.java:9056)
at javafx.scene.Node.applyCss(Node.java:9153)
at com.sun.javafx.scene.control.skin.VirtualFlow.setCellIndex(VirtualFlow.java:1964)
at com.sun.javafx.scene.control.skin.VirtualFlow.getCell(VirtualFlow.java:1797)
at com.sun.javafx.scene.control.skin.VirtualFlow.getCellLength(VirtualFlow.java:1879)
at com.sun.javafx.scene.control.skin.VirtualFlow.computeViewportOffset(VirtualFlow.java:2528)
at com.sun.javafx.scene.control.skin.VirtualFlow.layoutChildren(VirtualFlow.java:1189)
at javafx.scene.Parent.layout(Parent.java:1087)
at javafx.scene.Parent.layout(Parent.java:1093)
at javafx.scene.Parent.layout(Parent.java:1093)
at javafx.scene.Scene.doLayoutPass(Scene.java:552)
at javafx.scene.Scene$ScenePulseListener.pulse(Scene.java:2397)
at com.sun.javafx.tk.Toolkit.lambda$runPulse$29(Toolkit.java:398)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.tk.Toolkit.runPulse(Toolkit.java:397)
at com.sun.javafx.tk.Toolkit.firePulse(Toolkit.java:424)
at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:518)
at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:498)
at com.sun.javafx.tk.quantum.QuantumToolkit.pulseFromQueue(QuantumToolkit.java:491)
at com.sun.javafx.tk.quantum.QuantumToolkit.lambda$runToolkit$403(QuantumToolkit.java:319)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$147(WinApplication.java:177)
at java.lang.Thread.run(Thread.java:748)
Please try to either
change SimpleStringProperty(param.getValue().get(j).toString()) to
SimpleStringProperty(param.getValue().get(j-1).toString())
or
final int j = i; to final int j = i - 1;

Corda: Adding muliple output states in a single transaction

In our use case we have to add two output states with different contracts to a transaction. I tried to add multiple states to transaction in withItems. I call this flow from an RPC but it throws an exception as java.lang.IllegalArgumentException: List has more than one element. while signing the transaction.
This is how I am adding multiple output states:-
val outputstate1 = IouState1(dataModel1, me, otherParty)
val outputstate2 = IouState2(dataModel2, me, otherParty)
val cmd1 = Command(IouContract1.Commands.Create(), state1.participants.map { it.owningKey })
val cmd2 = Command(IouContract2.Commands.Initiated(), state2.participants.map { it.owningKey })
val txBuilder = TransactionBuilder(TransactionType.General, notary).withItems(state1, state2, cmd1, cmd2)
progressTracker.currentStep = VERIFYING_TRANSACTION
// Verifying the transaction.
txBuilder.toWireTransaction().toLedgerTransaction(serviceHub).verify()
progressTracker.currentStep = SIGNING_TRANSACTION
// Signing the transaction.
val partSignedTx = serviceHub.signInitialTransaction(txBuilder)
progressTracker.currentStep = GATHERING_SIGS
// Send the state to the counterparty, and receive it back with their signature.
val fullySignedTx = subFlow(CollectSignaturesFlow(partSignedTx, GATHERING_SIGS.childProgressTracker()))
// Finalising the transaction.
progressTracker.currentStep = FINALISING_TRANSACTION
// Notarise and record the transaction in both parties' vaults.
return subFlow(FinalityFlow(fullySignedTx, FINALISING_TRANSACTION.childProgressTracker())).single()
Stack trace:-
net.corda.core.flows.FlowException: java.lang.IllegalArgumentException: List has more than one element.
at net.corda.core.flows.SignTransactionFlow.call(CollectSignaturesFlow.kt:202) ~[corda-core-0.14.0.jar:?]
at net.corda.core.flows.SignTransactionFlow.call(CollectSignaturesFlow.kt:175) ~[corda-core-0.14.0.jar:?]
at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:166) ~[corda-core-0.14.0.jar:?]
at com.dwlabcorda.salescontract.LOCReceiverFlow.call(LOCContractFlow.kt:107) ~[tradefinancecorda-0.1.jar:?]
at com.dwlabcorda.salescontract.LOCReceiverFlow.call(LOCContractFlow.kt:95) ~[tradefinancecorda-0.1.jar:?]
at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:101) [corda-node-0.14.0.jar:?]
at net.corda.node.services.statemachine.FlowStateMachineImpl.run(FlowStateMachineImpl.kt:33) [corda-node-0.14.0.jar:?]
at co.paralleluniverse.fibers.Fiber.run1(Fiber.java:1067) [quasar-core-0.7.6-jdk8.jar:0.7.6]
at co.paralleluniverse.fibers.Fiber.exec(Fiber.java:767) [quasar-core-0.7.6-jdk8.jar:0.7.6]
at co.paralleluniverse.fibers.RunnableFiberTask.doExec(RunnableFiberTask.java:100) [quasar-core-0.7.6-jdk8.jar:0.7.6]
at co.paralleluniverse.fibers.RunnableFiberTask.run(RunnableFiberTask.java:91) [quasar-core-0.7.6-jdk8.jar:0.7.6]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.8.0_144]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:1.8.0_144]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [?:1.8.0_144]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_144]
at net.corda.node.utilities.AffinityExecutor$ServiceAffinityExecutor$1$thread$1.run(AffinityExecutor.kt:69) [corda-node-0.14.0.jar:?]
Caused by: java.lang.IllegalArgumentException: List has more than one element.
at kotlin.collections.CollectionsKt___CollectionsKt.single(_Collections.kt:471) ~[kotlin-stdlib-1.1.1.jar:1.1.1]
at com.dwlabcorda.salescontract.LOCReceiverFlow$call$signTransactionFlow$1.checkTransaction(LOCContractFlow.kt:100) ~[tradefinancecorda-0.1.jar:?]
at net.corda.core.flows.SignTransactionFlow.call(CollectSignaturesFlow.kt:199) ~[corda-core-0.14.0.jar:?]
... 17 more
Probably you used a .single() cast in one of your contracts that is now throwing an exception

How to map filenames to RDD using sc.textFile("s3n://bucket/*.csv")?

Please note, I must use the sc.textFile, but I would accept any other answers.
What I want to do is to simply add the filename that is being processed to RDD.... some thing like:
var rdd = sc.textFile("s3n://bucket/*.csv").map(line=>filename+","+line)
Much appreciated!
EDIT2: SOLUTION TO EDIT1 is to use Hadoop 2.4 or above. However, I have not tested it by using the slaves... etc. However, some of the mentioned solutions work only for the small data-sets. If you want to use the big-data, you will have to use the HadoopRDD
EDIT: I have tried the following, and it did not work:
:cp symjar/aws-java-sdk-1.9.29.jar
:cp symjar/aws-java-sdk-flow-build-tools-1.9.29.jar
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.{S3ObjectSummary, ObjectListing, GetObjectRequest}
import com.amazonaws.auth._
def awsAccessKeyId = "AKEY"
def awsSecretAccessKey = "SKEY"
val hadoopConf = sc.hadoopConfiguration;
hadoopConf.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3n.awsAccessKeyId", awsAccessKeyId)
hadoopConf.set("fs.s3n.awsSecretAccessKey", awsSecretAccessKey)
var rdd = sc.wholeTextFiles("s3n://bucket/dir/*.csv").map { case (filename, content) => filename }
rdd.count
NOTE: It is connecting to S3, and that is not an issue (as I have tested it many many times).
The error I get is:
INFO input.FileInputFormat: Total input paths to process : 4
java.io.FileNotFoundException: File does not exist: /RTLM-918/simple/t1-100.csv
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:517)
at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat$OneFileInfo.<init>(CombineFileInputFormat.java:489)
at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:280)
at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:240)
at org.apache.spark.rdd.WholeTextFileRDD.getPartitions(NewHadoopRDD.scala:267)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1511)
at org.apache.spark.rdd.RDD.collect(RDD.scala:813)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:34)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:36)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:44)
at $iwC$$iwC$$iwC.<init>(<console>:46)
at $iwC$$iwC.<init>(<console>:48)
at $iwC.<init>(<console>:50)
at <init>(<console>:52)
at .<init>(<console>:56)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
The only text method that includes the file name is wholeTextFiles.
sc.wholeTextFiles(path).map { case (filename, content) => ... }
Just pass it in as a variable during the map (or set as an object property).
sc.textFile("s3n://bucket/"+fn+".csv").map(line=>set_filepath(line,fn))
If you are dealing with big-data HadoopRDD is then the answer. Otherwise, with other suggestions, it won't work.
Code:
val text = sc.hadoopFile("s3n://.../", classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
// Cast to a HadoopRDD
val hadoopRdd = text.asInstanceOf[HadoopRDD[LongWritable, Text]]
val fileAndLine = hadoopRdd.mapPartitionsWithInputSplit { (inputSplit, iterator) ⇒
val file = inputSplit.asInstanceOf[FileSplit]
iterator.map { tpl ⇒ (file.getPath, tpl._2) }
}

Sonar-qube 4.5.1 custom rules plugin is not working

I have created a custom rule plugin using custom rule code snippet. Now I have added another rule which will show error when a anonymous class has more than 30 lines.
I have build the custom rule plugin jar using mvn package command and put in in sonar server extension/plugin and restarted the server.
my corresponding rule can be viewed from sonar server but rule cannot analyze the the code though there is a class with anonymous class with loc >30 lines. my custom rule is given below.
#Rule(
key = AnonymousClassesTooBigCheck.KEY,
name = "Avoid too big classes",
description = "avoid too bid classes",
tags = {"brain-overload"},
priority = Priority.MAJOR)
#BelongsToProfile(title = "Sonar way", priority = Priority.MAJOR)
public class AnonymousClassesTooBigCheck extends BaseTreeVisitor implements JavaFileScanner {
public static final String KEY = "AD0001";
private static final RuleKey RULE_KEY = RuleKey.of(MyJavaRulesDefinition.REPOSITORY_KEY, KEY);
private static final int DEFAULT_MAX = 30;
#RuleProperty(defaultValue = "" + DEFAULT_MAX)
public int max = DEFAULT_MAX;
private JavaFileScannerContext context;
/**
* Flag to skip check for class bodies of EnumConstants.
*/
private boolean isEnumConstantBody;
#Override
public void scanFile(JavaFileScannerContext context) {
this.context = context;
isEnumConstantBody = false;
scan(context.getTree());
}
#Override
public void visitNewClass(NewClassTree tree) {
if (tree.classBody() != null && !isEnumConstantBody) {
int lines = getNumberOfLines(tree.classBody());
if (lines > max) {
context.addIssue(tree, RULE_KEY, "Reduce this anonymous class number of lines from " + lines + " to at most " + max + ", or make it a named class.");
}
}
isEnumConstantBody = false;
super.visitNewClass(tree);
}
#Override
public void visitEnumConstant(EnumConstantTree tree) {
isEnumConstantBody = true;
super.visitEnumConstant(tree);
}
#Override
public void visitLambdaExpression(LambdaExpressionTree lambdaExpressionTree) {
int lines = getNumberOfLines(((JavaTree) lambdaExpressionTree.body()).getAstNode());
if (lines > max) {
context.addIssue(lambdaExpressionTree, RULE_KEY, "Reduce this lambda expression number of lines from " + lines + " to at most " + max + ".");
}
super.visitLambdaExpression(lambdaExpressionTree);
}
private int getNumberOfLines(ClassTree classTree) {
int startLine = ((InternalSyntaxToken) classTree.openBraceToken()).getLine();
int endline = ((InternalSyntaxToken) classTree.closeBraceToken()).getLine();
return endline - startLine + 1;
}
private int getNumberOfLines(AstNode node) {
return node.getLastToken().getLine() - node.getTokenLine() + 1;
}
}
Now do I need to do any thing more to get the rule hit?
One more thing
after putting the plugin jar in plugin folder if i run the analyzer using mvn sonar:sonar a error occures
Embedded error: org/sonar/java/model/InternalSyntaxToken
org.sonar.java.model.InternalSyntaxToken
[INFO] ------------------------------------------------------------------------
[INFO] Trace
org.apache.maven.lifecycle.LifecycleExecutionException: Can not execute Sonar
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:719)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:569)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:539)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:284)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
Caused by: org.apache.maven.plugin.MojoExecutionException: Can not execute Sonar
at org.codehaus.mojo.sonar.Bootstraper.executeMojo(Bootstraper.java:103)
at org.codehaus.mojo.sonar.Bootstraper.start(Bootstraper.java:79)
at org.codehaus.mojo.sonar.SonarMojo.execute(SonarMojo.java:88)
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
... 17 more
Caused by: org.apache.maven.plugin.MojoExecutionException: org/sonar/java/model/InternalSyntaxToken
at org.sonar.maven.ExceptionHandling.handle(ExceptionHandling.java:37)
at org.sonar.maven.SonarMojo.execute(SonarMojo.java:175)
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
at org.codehaus.mojo.sonar.Bootstraper.executeMojo(Bootstraper.java:98)
... 21 more
Caused by: java.lang.NoClassDefFoundError: org/sonar/java/model/InternalSyntaxToken
at org.sonar.samples.java.AnonymousClassesTooBigCheck.getNumberOfLines(AnonymousClassesTooBigCheck.java:78)
at org.sonar.samples.java.AnonymousClassesTooBigCheck.visitNewClass(AnonymousClassesTooBigCheck.java:53)
at org.sonar.java.model.expression.NewClassTreeImpl.accept(NewClassTreeImpl.java:126)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.scan(BaseTreeVisitor.java:42)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.visitVariable(BaseTreeVisitor.java:291)
at org.sonar.java.model.declaration.VariableTreeImpl.accept(VariableTreeImpl.java:180)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.scan(BaseTreeVisitor.java:42)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.scan(BaseTreeVisitor.java:36)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.visitClass(BaseTreeVisitor.java:69)
at org.sonar.java.model.declaration.ClassTreeImpl.accept(ClassTreeImpl.java:201)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.scan(BaseTreeVisitor.java:42)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.scan(BaseTreeVisitor.java:36)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.visitClass(BaseTreeVisitor.java:69)
at org.sonar.java.model.declaration.ClassTreeImpl.accept(ClassTreeImpl.java:201)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.scan(BaseTreeVisitor.java:42)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.scan(BaseTreeVisitor.java:36)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.visitCompilationUnit(BaseTreeVisitor.java:55)
at org.sonar.java.model.JavaTree$CompilationUnitTreeImpl.accept(JavaTree.java:202)
at org.sonar.plugins.java.api.tree.BaseTreeVisitor.scan(BaseTreeVisitor.java:42)
at org.sonar.samples.java.AnonymousClassesTooBigCheck.scanFile(AnonymousClassesTooBigCheck.java:47)
at org.sonar.java.model.VisitorsBridge.visitFile(VisitorsBridge.java:123)
at com.sonar.sslr.impl.ast.AstWalker.walkAndVisit(AstWalker.java:67)
at org.sonar.java.ast.AstScanner.simpleScan(AstScanner.java:107)
at org.sonar.java.ast.AstScanner.scan(AstScanner.java:75)
at org.sonar.java.JavaSquid.scanSources(JavaSquid.java:122)
at org.sonar.java.JavaSquid.scan(JavaSquid.java:115)
at org.sonar.plugins.java.JavaSquidSensor.analyse(JavaSquidSensor.java:91)
at org.sonar.batch.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:79)
at org.sonar.batch.phases.SensorsExecutor.execute(SensorsExecutor.java:70)
at org.sonar.batch.phases.PhaseExecutor.execute(PhaseExecutor.java:119)
at org.sonar.batch.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:194)
at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:92)
at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
at org.sonar.batch.scan.ProjectScanContainer.scan(ProjectScanContainer.java:233)
at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:228)
at org.sonar.batch.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:221)
at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:92)
at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
at org.sonar.batch.scan.ScanTask.scan(ScanTask.java:64)
at org.sonar.batch.scan.ScanTask.execute(ScanTask.java:51)
at org.sonar.batch.bootstrap.TaskContainer.doAfterStart(TaskContainer.java:125)
at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:92)
at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
at org.sonar.batch.bootstrap.BootstrapContainer.executeTask(BootstrapContainer.java:173)
at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:95)
at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:67)
at org.sonar.runner.batch.IsolatedLauncher.execute(IsolatedLauncher.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:87)
at org.sonar.runner.impl.BatchLauncher$1.run(BatchLauncher.java:75)
at java.security.AccessController.doPrivileged(Native Method)
at org.sonar.runner.impl.BatchLauncher.doExecute(BatchLauncher.java:69)
at org.sonar.runner.impl.BatchLauncher.execute(BatchLauncher.java:50)
at org.sonar.runner.api.EmbeddedRunner.doExecute(EmbeddedRunner.java:102)
at org.sonar.runner.api.Runner.execute(Runner.java:100)
at org.sonar.maven.SonarMojo.execute(SonarMojo.java:173)
... 23 more
Caused by: java.lang.ClassNotFoundException: org.sonar.java.model.InternalSyntaxToken
at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:259)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:235)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:227)
... 82 more
The reason you encounter this error is that the code of the AnonymousClassesTooBigCheck is using internal APIs, here InternalSyntaxToken is not supposed to be used directly and thus fails at runtime (you might notice that none of the example in the custom rule code snippet is using this class).

Why am i getting this error java.lang.ClassNotFoundException?

Why am i getting this error message?
Testcase: createIndexBatch_usingTheTestArchive(src.LireHandlerTest): Caused an ERROR
at/lux/imageanalysis/ColorLayoutImpl
java.lang.NoClassDefFoundError: at/lux/imageanalysis/ColorLayoutImpl
at net.semanticmetadata.lire.impl.SimpleDocumentBuilder.createDocument(Unknown Source)
at net.semanticmetadata.lire.AbstractDocumentBuilder.createDocument(Unknown Source)
at backend.storage.LireHandler.createIndexBatch(LireHandler.java:49)
at backend.storage.LireHandler.createIndexBatch(LireHandler.java:57)
at src.LireHandlerTest.createIndexBatch_usingTheTestArchive(LireHandlerTest.java:56)
Caused by: java.lang.ClassNotFoundException: at.lux.imageanalysis.ColorLayoutImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
I am trying to create a Lucene index using documents that are created using Lire. When i am getting to the point where it tries to create a document using the document builder it gives me this error message.
input paramaters first run:
filepath: "test_archive" (this is a image archive)
whereToStoreIndex: "test_index" (Where i am going to store the index)
createNewIndex: true
Since the method is recursive (see the if statement where it checks if is a directory) it will call itself multiple times but all the recursive calls uses createNewIndex = false.
Here is the code:
public static boolean createIndexBatch(String filepath, String whereToStoreIndex, boolean createNewIndex){
DocumentBuilder docBldr = DocumentBuilderFactory.getDefaultDocumentBuilder();
try{
IndexWriter indexWriter = new IndexWriter(
FSDirectory.open(new File(whereToStoreIndex)),
new SimpleAnalyzer(),
createNewIndex,
IndexWriter.MaxFieldLength.UNLIMITED
);
File[] files = FileHandler.getFilesFromDirectory(filepath);
for(File f:files){
if(f.isFile()){
//Hopper over Thumbs.db filene...
if(!f.getName().equals("Thumbs.db")){
//Creating the document that is going to be stored in the index
String name = f.getName();
String path = f.getPath();
FileInputStream stream = new FileInputStream(f);
Document doc = docBldr.createDocument(stream, f.getName());
//Add document to the index
indexWriter.addDocument(doc);
}
}else if(f.isDirectory()){
indexWriter.optimize();
indexWriter.close();
LireHandler.createIndexBatch(f.getPath(), whereToStoreIndex, false);
}
}
indexWriter.close();
}catch(IOException e){
System.out.print("IOException in createIndexBatch:\n"+e.getMessage()+"\n");
return false;
}
return true;
}
It sounds like you are missing a java library.
I think you need to download Caliph & Emir, that may be used indirectly by Lire.
http://www.semanticmetadata.net/download/