Most efficient way to deal with ORA-01795:maximum number of expressions in a list is 1000 in hibernate - sql

I have to perform a select on which I have more than 1000 elements via hibernate, and then I received the error "ORA-01795:maximum number of expressions in a list is 1000" when I'm using the Oracle brand.
SELECT * FROM matable WHERE column IN (?,?,...) (>1000 items)
I found many solutions :
Split the list with OR
where A in (a,b,c,d,e,f)
becomes
where (A in (a,b,c) OR a in (d,e,f)) ...
Create a table with UNION ALL
SELECT * FROM maintable
JOIN (
SELECT v1 a FROM DUAL UNION ALL
SELECT v2 a FROM DUAL UNION ALL
SELECT v3 a FROM DUAL UNION ALL
...
SELECT v2000 a FROM DUAL) tmp
on tmp.a = maintable.id
Using tuples to get rid of the limit
where (column,0) in ((1,0),(2,0),(3,0),(4,0), ... ,(1500,0))
Using a temporary table..
where A in SELECT item FROM my_temporary_table
References here and there and also there.
My question is the following : what is the best practice to deal with this issue? By best practice I mean the most performant, but not only for Oracle; if I use hibernate, I don't want to create and manage a different code for each brand of database (I'm concerned by Oracle, MS SQL and PostGre only).
My first reaction would have been to use a temporary table, but I don't know what has the most impact.

Use a temporary table and make the values primary keys on the table. This should allow very efficient optimizations for comparison. The most like is simply an index lookup, although if the table is very small, Oracle might choose some other method such as a table scan.
This method should be faster than 1,000 or conditions, in almost any database. Sometimes in is optimized in a similar way (using a binary tree to store the values). In such databases, the performance would be similar.

I fixed this issue with some changes in hibernate-core jar.
I made a helper class to split an expression in more joins like: ... t.column IN (: list_1) OR t.column IN (: list_2) ... , Then I changed AbstractQueryImpl.expandParameterList method from hibernate to call my method if the collection exceeds the limit.
My hibernate-core version is 3.6.10.Final, but it work fine and for 4.x versions - I tested it.
My code is tested for next cases:
where t.id in (:idList)
where (t.id in (:idList))
where ((t.id) in (:idList))
where 1=1 and t.id in (:idList)
where 1=1 and (t.id in (:idList))
where 1=1 and(t.id) in (:idList)
where 1=1 and((t.id) in (:idList))
where 1=1 and(t.id in (:idList))
where t.id not in (:idList)
where (t.id not in (:idList))
where ((t.id) not in (:idList))
AbstractQueryImpl.expandParameterList :
private String expandParameterList(String query, String name, TypedValue typedList, Map namedParamsCopy) {
Collection vals = (Collection) typedList.getValue();
Type type = typedList.getType();
boolean isJpaPositionalParam = parameterMetadata.getNamedParameterDescriptor( name ).isJpaStyle();
String paramPrefix = isJpaPositionalParam ? "?" : ParserHelper.HQL_VARIABLE_PREFIX;
String placeholder =
new StringBuffer( paramPrefix.length() + name.length() )
.append( paramPrefix ).append( name )
.toString();
if ( query == null ) {
return query;
}
int loc = query.indexOf( placeholder );
if ( loc < 0 ) {
return query;
}
String beforePlaceholder = query.substring( 0, loc );
String afterPlaceholder = query.substring( loc + placeholder.length() );
// check if placeholder is already immediately enclosed in parentheses
// (ignoring whitespace)
boolean isEnclosedInParens =
StringHelper.getLastNonWhitespaceCharacter( beforePlaceholder ) == '(' &&
StringHelper.getFirstNonWhitespaceCharacter( afterPlaceholder ) == ')';
if ( vals.size() == 1 && isEnclosedInParens ) {
// short-circuit for performance when only 1 value and the
// placeholder is already enclosed in parentheses...
namedParamsCopy.put( name, new TypedValue( type, vals.iterator().next(), session.getEntityMode() ) );
return query;
}
// *** changes by Vasile Bors for HHH-1123 ***
// case vals.size() > 1000
if ((vals.size() >= InExpressionExpander.MAX_ALLOWED_PER_INEXPR) && isEnclosedInParens) {
InExpressionExpander inExpressionExpander = new InExpressionExpander(beforePlaceholder, afterPlaceholder);
if(inExpressionExpander.isValidInOrNotInExpression()){
List<String> list = new ArrayList<String>( vals.size() );
Iterator iter = vals.iterator();
int i = 0;
String alias;
while ( iter.hasNext() ) {
alias = ( isJpaPositionalParam ? 'x' + name : name ) + i++ + '_';
namedParamsCopy.put( alias, new TypedValue( type, iter.next(), session.getEntityMode() ) );
list.add(ParserHelper.HQL_VARIABLE_PREFIX + alias );
}
String expandedExpression = inExpressionExpander.expandExpression(list);
if(expandedExpression != null){
return expandedExpression;
}
}
}
// *** end changes by Vasile Bors for HHH-1123 ***
StringBuffer list = new StringBuffer(16);
Iterator iter = vals.iterator();
int i = 0;
while (iter.hasNext()) {
String alias = (isJpaPositionalParam ? 'x' + name : name) + i++ + '_';
namedParamsCopy.put(alias, new TypedValue(type, iter.next(), session.getEntityMode()));
list.append(ParserHelper.HQL_VARIABLE_PREFIX).append(alias);
if (iter.hasNext()) {
list.append(", ");
}
}
return StringHelper.replace(
beforePlaceholder,
afterPlaceholder,
placeholder.toString(),
list.toString(),
true,
true
);
}
My helper class InExpressionExpander:
package org.hibernate.util;
import org.hibernate.QueryException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Iterator;
import java.util.List;
import java.util.Stack;
/**
* Utility class for expand Hql and Sql IN expressions with a parameter with more than IN expression limit size (HHH-1123).
* <br/>
* It work for expression with formats:
* <pre>
*
* where t.id in (:idList)
* where (t.id in (:idList))
* where ((t.id) in (:idList))
* where 1=1 and t.id in (:idList)
* where 1=1 and (t.id in (:idList))
* where 1=1 and(t.id) in (:idList)
* where 1=1 and((t.id) in (:idList))
* where 1=1 and(t.id in (:idList))
*
* where t.id not in (:idList)
* where (t.id not in (:idList))
* where ((t.id) not in (:idList))
* </pre>
* <p/>
* Example:
* <pre>
* select t.id from tableOrEntity t where t.id IN (:idList)
* </pre
*
* #author Vasile Bors
* #since 13/12/2015.
*/
public class InExpressionExpander {
private static final Logger log = LoggerFactory.getLogger(InExpressionExpander.class);
public static final int MAX_ALLOWED_PER_INEXPR = 1000;
private static final int MAX_PARAMS_PER_INEXPR = 500;
private Stack<String> stackExpr = new Stack<String>();
private StringBuilder toWalkQuery;
private final String beforePlaceholder;
private final String afterPlaceholder;
private boolean wasChecked = false;
private boolean isEnclosedInParens = false;
private boolean isInExpr = false;
private boolean isNotInExpr = false;
public InExpressionExpander(String beforePlaceholder, String afterPlaceholder) {
this.toWalkQuery = new StringBuilder(beforePlaceholder);
this.beforePlaceholder = beforePlaceholder;
this.afterPlaceholder = afterPlaceholder;
}
public boolean isValidInOrNotInExpression() {
if (!wasChecked) {
String lastExpr = extractLastExpression();
if ("(".equals(lastExpr)) {
isEnclosedInParens = true;
lastExpr = extractLastExpression();
}
isInExpr = "in".equalsIgnoreCase(lastExpr);
}
wasChecked = true;
return isInExpr;
}
public String expandExpression(List paramList) {
if (isValidInOrNotInExpression()) {
final String lastExpr = extractLastExpression(false);
if ("not".equalsIgnoreCase(lastExpr)) {
isNotInExpr = true;
extractLastExpression(); //extract "not" and consume it
}
extractColumnForInExpression();
StringBuilder exprPrefixBuilder = new StringBuilder();
for (int i = stackExpr.size() - 1; i > -1; i--) {
exprPrefixBuilder.append(stackExpr.get(i)).append(' ');
}
if (!isEnclosedInParens) {
exprPrefixBuilder.append('(');
}
String expandedExpression = expandInExpression(exprPrefixBuilder, paramList);
String beforeExpression = getBeforeExpression();
String afterExpression = getAfterExpression();
String expandedQuery = new StringBuilder(beforeExpression).append(expandedExpression)
.append(afterExpression)
.toString();
if (log.isDebugEnabled()) {
log.debug(
"Query was changed to prevent exception for maximum number of expression in a list. Expanded IN expression query:\n {}",
expandedExpression);
log.debug("Expanded query:\n {}", expandedQuery);
}
return expandedQuery;
}
log.error("Illegal call of InExpressionExpander.expandExpression() without IN expression.");
return null;
}
private String expandInExpression(StringBuilder exprPrefixBuilder, List values) {
String joinExpr = isNotInExpr ? ") and " : ") or ";
StringBuilder expr = new StringBuilder(16);
Iterator iter = values.iterator();
int i = 0;
boolean firstExpr = true;
while (iter.hasNext()) {
if (firstExpr || i % MAX_PARAMS_PER_INEXPR == 0) {
//close previous expression and start new expression
if (!firstExpr) {
expr.append(joinExpr);
} else {
firstExpr = false;
}
expr.append(exprPrefixBuilder);
} else {
expr.append(", ");
}
expr.append(iter.next());
i++;
}
expr.append(')');// close for last in expression
return expr.toString();
}
/**
* Method extract last expression parsed by space from toWalkQuery and remove it from toWalkQuery;<br/>
* If expression has brackets it will return al content between brackets and it will add additional space to adjust splitting by space.
*
* #return last expression from toWalkQuery
*/
private String extractLastExpression() {
return extractLastExpression(true);
}
/**
* Method extract last expression parsed by space from toWalkQuery, remove it from toWalkQuery if is consume = true;<br/>
* If expression has brackets it will return al content between brackets and it will add additional space to adjust splitting by space.
*
* #param consum if true the method will extract and remove last expression from toWalkQuery
* #return last expression from toWalkQuery
*/
private String extractLastExpression(final boolean consum) {
int lastIndex = this.toWalkQuery.length() - 1;
String lastExpr;
int exprSeparatorIndex = this.toWalkQuery.lastIndexOf(" ");
if (lastIndex == exprSeparatorIndex) { //remove last space from the end
this.toWalkQuery.delete(exprSeparatorIndex, this.toWalkQuery.length());
return extractLastExpression(consum);
} else {
lastExpr = this.toWalkQuery.substring(exprSeparatorIndex + 1, this.toWalkQuery.length());
if (lastExpr.length() > 1) {
if (lastExpr.endsWith(")")) {
//if parens are closed at the end we need to find where it is open
int opensParens = 0;
int closedParens = 0;
int startExprIndex = -1;
char c;
for (int i = lastExpr.length() - 1; i > -1; i--) {
c = lastExpr.charAt(i);
if (c == ')') {
closedParens++;
} else if (c == '(') {
opensParens++;
}
if (closedParens == opensParens) {
startExprIndex = i;
break;
}
}
if (startExprIndex > -1) {
lastExpr = lastExpr.substring(startExprIndex, lastExpr.length());
exprSeparatorIndex = exprSeparatorIndex + startExprIndex
+ 1; // +1 because separator is not space and don't must be deleted
}
} else if (lastExpr.contains("(")) {
int parentsIndex = exprSeparatorIndex + lastExpr.indexOf('(') + 1;
this.toWalkQuery.replace(parentsIndex, parentsIndex + 1, " ( ");
return extractLastExpression(consum);
}
}
if (consum) {
this.toWalkQuery.delete(exprSeparatorIndex, this.toWalkQuery.length());
}
}
if (consum) {
stackExpr.push(lastExpr);
}
return lastExpr;
}
private String extractColumnForInExpression() {
String column = extractLastExpression();
String beforeColumn = extractLastExpression(false);
long pointIndx = beforeColumn.lastIndexOf('.');
if (pointIndx > -1) {
if (pointIndx == (beforeColumn.length() - 1)) {
throw new QueryException(
"Invalid column format: " + beforeColumn + ' ' + column
+ " . Remove space from column!");
}
}
return column;
}
private String getBeforeExpression() {
return this.toWalkQuery + " (";
}
private String getAfterExpression() {
if (StringHelper.getFirstNonWhitespaceCharacter(afterPlaceholder) == ')') {
return afterPlaceholder;
}
return afterPlaceholder + ") ";
}
}
I am happy to receive any suggestions for improving this solution.

Related

Arithmetic parser in kotlin [duplicate]

I'm trying to write a Java routine to evaluate math expressions from String values like:
"5+3"
"10-4*5"
"(1+10)*3"
I want to avoid a lot of if-then-else statements.
How can I do this?
With JDK1.6, you can use the built-in Javascript engine.
import javax.script.ScriptEngineManager;
import javax.script.ScriptEngine;
import javax.script.ScriptException;
public class Test {
public static void main(String[] args) throws ScriptException {
ScriptEngineManager mgr = new ScriptEngineManager();
ScriptEngine engine = mgr.getEngineByName("JavaScript");
String foo = "40+2";
System.out.println(engine.eval(foo));
}
}
I've written this eval method for arithmetic expressions to answer this question. It does addition, subtraction, multiplication, division, exponentiation (using the ^ symbol), and a few basic functions like sqrt. It supports grouping using (...), and it gets the operator precedence and associativity rules correct.
public static double eval(final String str) {
return new Object() {
int pos = -1, ch;
void nextChar() {
ch = (++pos < str.length()) ? str.charAt(pos) : -1;
}
boolean eat(int charToEat) {
while (ch == ' ') nextChar();
if (ch == charToEat) {
nextChar();
return true;
}
return false;
}
double parse() {
nextChar();
double x = parseExpression();
if (pos < str.length()) throw new RuntimeException("Unexpected: " + (char)ch);
return x;
}
// Grammar:
// expression = term | expression `+` term | expression `-` term
// term = factor | term `*` factor | term `/` factor
// factor = `+` factor | `-` factor | `(` expression `)` | number
// | functionName `(` expression `)` | functionName factor
// | factor `^` factor
double parseExpression() {
double x = parseTerm();
for (;;) {
if (eat('+')) x += parseTerm(); // addition
else if (eat('-')) x -= parseTerm(); // subtraction
else return x;
}
}
double parseTerm() {
double x = parseFactor();
for (;;) {
if (eat('*')) x *= parseFactor(); // multiplication
else if (eat('/')) x /= parseFactor(); // division
else return x;
}
}
double parseFactor() {
if (eat('+')) return +parseFactor(); // unary plus
if (eat('-')) return -parseFactor(); // unary minus
double x;
int startPos = this.pos;
if (eat('(')) { // parentheses
x = parseExpression();
if (!eat(')')) throw new RuntimeException("Missing ')'");
} else if ((ch >= '0' && ch <= '9') || ch == '.') { // numbers
while ((ch >= '0' && ch <= '9') || ch == '.') nextChar();
x = Double.parseDouble(str.substring(startPos, this.pos));
} else if (ch >= 'a' && ch <= 'z') { // functions
while (ch >= 'a' && ch <= 'z') nextChar();
String func = str.substring(startPos, this.pos);
if (eat('(')) {
x = parseExpression();
if (!eat(')')) throw new RuntimeException("Missing ')' after argument to " + func);
} else {
x = parseFactor();
}
if (func.equals("sqrt")) x = Math.sqrt(x);
else if (func.equals("sin")) x = Math.sin(Math.toRadians(x));
else if (func.equals("cos")) x = Math.cos(Math.toRadians(x));
else if (func.equals("tan")) x = Math.tan(Math.toRadians(x));
else throw new RuntimeException("Unknown function: " + func);
} else {
throw new RuntimeException("Unexpected: " + (char)ch);
}
if (eat('^')) x = Math.pow(x, parseFactor()); // exponentiation
return x;
}
}.parse();
}
Example:
System.out.println(eval("((4 - 2^3 + 1) * -sqrt(3*3+4*4)) / 2"));
Output: 7.5 (which is correct)
The parser is a recursive descent parser, so internally uses separate parse methods for each level of operator precedence in its grammar. I deliberately kept it short, but here are some ideas you might want to expand it with:
Variables:
The bit of the parser that reads the names for functions can easily be changed to handle custom variables too, by looking up names in a variable table passed to the eval method, such as a Map<String,Double> variables.
Separate compilation and evaluation:
What if, having added support for variables, you wanted to evaluate the same expression millions of times with changed variables, without parsing it every time? It's possible. First define an interface to use to evaluate the precompiled expression:
#FunctionalInterface
interface Expression {
double eval();
}
Now to rework the original "eval" function into a "parse" function, change all the methods that return doubles, so instead they return an instance of that interface. Java 8's lambda syntax works well for this. Example of one of the changed methods:
Expression parseExpression() {
Expression x = parseTerm();
for (;;) {
if (eat('+')) { // addition
Expression a = x, b = parseTerm();
x = (() -> a.eval() + b.eval());
} else if (eat('-')) { // subtraction
Expression a = x, b = parseTerm();
x = (() -> a.eval() - b.eval());
} else {
return x;
}
}
}
That builds a recursive tree of Expression objects representing the compiled expression (an abstract syntax tree). Then you can compile it once and evaluate it repeatedly with different values:
public static void main(String[] args) {
Map<String,Double> variables = new HashMap<>();
Expression exp = parse("x^2 - x + 2", variables);
for (double x = -20; x <= +20; x++) {
variables.put("x", x);
System.out.println(x + " => " + exp.eval());
}
}
Different datatypes:
Instead of double, you could change the evaluator to use something more powerful like BigDecimal, or a class that implements complex numbers, or rational numbers (fractions). You could even use Object, allowing some mix of datatypes in expressions, just like a real programming language. :)
All code in this answer released to the public domain. Have fun!
For my university project, I was looking for a parser / evaluator supporting both basic formulas and more complicated equations (especially iterated operators). I found very nice open source library for JAVA and .NET called mXparser. I will give a few examples to make some feeling on the syntax, for further instructions please visit project website (especially tutorial section).
https://mathparser.org/
https://mathparser.org/mxparser-tutorial/
https://mathparser.org/api/
And few examples
1 - Simple furmula
Expression e = new Expression("( 2 + 3/4 + sin(pi) )/2");
double v = e.calculate()
2 - User defined arguments and constants
Argument x = new Argument("x = 10");
Constant a = new Constant("a = pi^2");
Expression e = new Expression("cos(a*x)", x, a);
double v = e.calculate()
3 - User defined functions
Function f = new Function("f(x, y, z) = sin(x) + cos(y*z)");
Expression e = new Expression("f(3,2,5)", f);
double v = e.calculate()
4 - Iteration
Expression e = new Expression("sum( i, 1, 100, sin(i) )");
double v = e.calculate()
Found recently - in case you would like to try the syntax (and see the advanced use case) you can download the Scalar Calculator app that is powered by mXparser.
The correct way to solve this is with a lexer and a parser. You can write simple versions of these yourself, or those pages also have links to Java lexers and parsers.
Creating a recursive descent parser is a really good learning exercise.
HERE is another open source library on GitHub named EvalEx.
Unlike the JavaScript engine this library is focused in evaluating mathematical expressions only. Moreover, the library is extensible and supports use of boolean operators as well as parentheses.
You can evaluate expressions easily if your Java application already accesses a database, without using any other JARs.
Some databases require you to use a dummy table (eg, Oracle's "dual" table) and others will allow you to evaluate expressions without "selecting" from any table.
For example, in Sql Server or Sqlite
select (((12.10 +12.0))/ 233.0) amount
and in Oracle
select (((12.10 +12.0))/ 233.0) amount from dual;
The advantage of using a DB is that you can evaluate many expressions at the same time. Also most DB's will allow you to use highly complex expressions and will also have a number of extra functions that can be called as necessary.
However performance may suffer if many single expressions need to be evaluated individually, particularly when the DB is located on a network server.
The following addresses the performance problem to some extent, by using a Sqlite in-memory database.
Here's a full working example in Java
Class. forName("org.sqlite.JDBC");
Connection conn = DriverManager.getConnection("jdbc:sqlite::memory:");
Statement stat = conn.createStatement();
ResultSet rs = stat.executeQuery( "select (1+10)/20.0 amount");
rs.next();
System.out.println(rs.getBigDecimal(1));
stat.close();
conn.close();
Of course you could extend the above code to handle multiple calculations at the same time.
ResultSet rs = stat.executeQuery( "select (1+10)/20.0 amount, (1+100)/20.0 amount2");
You can also try the BeanShell interpreter:
Interpreter interpreter = new Interpreter();
interpreter.eval("result = (7+21*6)/(32-27)");
System.out.println(interpreter.get("result"));
Another way is to use the Spring Expression Language or SpEL which does a whole lot more along with evaluating mathematical expressions, therefore maybe slightly overkill. You do not have to be using Spring framework to use this expression library as it is stand-alone. Copying examples from SpEL's documentation:
ExpressionParser parser = new SpelExpressionParser();
int two = parser.parseExpression("1 + 1").getValue(Integer.class); // 2
double twentyFour = parser.parseExpression("2.0 * 3e0 * 4").getValue(Double.class); //24.0
This article discusses various approaches. Here are the 2 key approaches mentioned in the article:
JEXL from Apache
Allows for scripts that include references to java objects.
// Create or retrieve a JexlEngine
JexlEngine jexl = new JexlEngine();
// Create an expression object
String jexlExp = "foo.innerFoo.bar()";
Expression e = jexl.createExpression( jexlExp );
// Create a context and add data
JexlContext jctx = new MapContext();
jctx.set("foo", new Foo() );
// Now evaluate the expression, getting the result
Object o = e.evaluate(jctx);
Use the javascript engine embedded in the JDK:
private static void jsEvalWithVariable()
{
List<String> namesList = new ArrayList<String>();
namesList.add("Jill");
namesList.add("Bob");
namesList.add("Laureen");
namesList.add("Ed");
ScriptEngineManager mgr = new ScriptEngineManager();
ScriptEngine jsEngine = mgr.getEngineByName("JavaScript");
jsEngine.put("namesListKey", namesList);
System.out.println("Executing in script environment...");
try
{
jsEngine.eval("var x;" +
"var names = namesListKey.toArray();" +
"for(x in names) {" +
" println(names[x]);" +
"}" +
"namesListKey.add(\"Dana\");");
}
catch (ScriptException ex)
{
ex.printStackTrace();
}
}
if we are going to implement it then we can can use the below algorithm :--
While there are still tokens to be read in,
1.1 Get the next token.
1.2 If the token is:
1.2.1 A number: push it onto the value stack.
1.2.2 A variable: get its value, and push onto the value stack.
1.2.3 A left parenthesis: push it onto the operator stack.
1.2.4 A right parenthesis:
1 While the thing on top of the operator stack is not a
left parenthesis,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
2 Pop the left parenthesis from the operator stack, and discard it.
1.2.5 An operator (call it thisOp):
1 While the operator stack is not empty, and the top thing on the
operator stack has the same or greater precedence as thisOp,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
2 Push thisOp onto the operator stack.
While the operator stack is not empty,
1 Pop the operator from the operator stack.
2 Pop the value stack twice, getting two operands.
3 Apply the operator to the operands, in the correct order.
4 Push the result onto the value stack.
At this point the operator stack should be empty, and the value
stack should have only one value in it, which is the final result.
This is another interesting alternative
https://github.com/Shy-Ta/expression-evaluator-demo
The usage is very simple and gets the job done, for example:
ExpressionsEvaluator evalExpr = ExpressionsFactory.create("2+3*4-6/2");
assertEquals(BigDecimal.valueOf(11), evalExpr.eval());
It seems like JEP should do the job
It's too late to answer but I came across same situation to evaluate expression in java, it might help someone
MVEL does runtime evaluation of expressions, we can write a java code in String to get it evaluated in this.
String expressionStr = "x+y";
Map<String, Object> vars = new HashMap<String, Object>();
vars.put("x", 10);
vars.put("y", 20);
ExecutableStatement statement = (ExecutableStatement) MVEL.compileExpression(expressionStr);
Object result = MVEL.executeExpression(statement, vars);
Try the following sample code using JDK1.6's Javascript engine with code injection handling.
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
public class EvalUtil {
private static ScriptEngine engine = new ScriptEngineManager().getEngineByName("JavaScript");
public static void main(String[] args) {
try {
System.out.println((new EvalUtil()).eval("(((5+5)/2) > 5) || 5 >3 "));
System.out.println((new EvalUtil()).eval("(((5+5)/2) > 5) || true"));
} catch (Exception e) {
e.printStackTrace();
}
}
public Object eval(String input) throws Exception{
try {
if(input.matches(".*[a-zA-Z;~`#$_{}\\[\\]:\\\\;\"',\\.\\?]+.*")) {
throw new Exception("Invalid expression : " + input );
}
return engine.eval(input);
} catch (Exception e) {
e.printStackTrace();
throw e;
}
}
}
This is actually complementing the answer given by #Boann. It has a slight bug which causes "-2 ^ 2" to give an erroneous result of -4.0. The problem for that is the point at which the exponentiation is evaluated in his. Just move the exponentiation to the block of parseTerm(), and you'll be all fine. Have a look at the below, which is #Boann's answer slightly modified. Modification is in the comments.
public static double eval(final String str) {
return new Object() {
int pos = -1, ch;
void nextChar() {
ch = (++pos < str.length()) ? str.charAt(pos) : -1;
}
boolean eat(int charToEat) {
while (ch == ' ') nextChar();
if (ch == charToEat) {
nextChar();
return true;
}
return false;
}
double parse() {
nextChar();
double x = parseExpression();
if (pos < str.length()) throw new RuntimeException("Unexpected: " + (char)ch);
return x;
}
// Grammar:
// expression = term | expression `+` term | expression `-` term
// term = factor | term `*` factor | term `/` factor
// factor = `+` factor | `-` factor | `(` expression `)`
// | number | functionName factor | factor `^` factor
double parseExpression() {
double x = parseTerm();
for (;;) {
if (eat('+')) x += parseTerm(); // addition
else if (eat('-')) x -= parseTerm(); // subtraction
else return x;
}
}
double parseTerm() {
double x = parseFactor();
for (;;) {
if (eat('*')) x *= parseFactor(); // multiplication
else if (eat('/')) x /= parseFactor(); // division
else if (eat('^')) x = Math.pow(x, parseFactor()); //exponentiation -> Moved in to here. So the problem is fixed
else return x;
}
}
double parseFactor() {
if (eat('+')) return parseFactor(); // unary plus
if (eat('-')) return -parseFactor(); // unary minus
double x;
int startPos = this.pos;
if (eat('(')) { // parentheses
x = parseExpression();
eat(')');
} else if ((ch >= '0' && ch <= '9') || ch == '.') { // numbers
while ((ch >= '0' && ch <= '9') || ch == '.') nextChar();
x = Double.parseDouble(str.substring(startPos, this.pos));
} else if (ch >= 'a' && ch <= 'z') { // functions
while (ch >= 'a' && ch <= 'z') nextChar();
String func = str.substring(startPos, this.pos);
x = parseFactor();
if (func.equals("sqrt")) x = Math.sqrt(x);
else if (func.equals("sin")) x = Math.sin(Math.toRadians(x));
else if (func.equals("cos")) x = Math.cos(Math.toRadians(x));
else if (func.equals("tan")) x = Math.tan(Math.toRadians(x));
else throw new RuntimeException("Unknown function: " + func);
} else {
throw new RuntimeException("Unexpected: " + (char)ch);
}
//if (eat('^')) x = Math.pow(x, parseFactor()); // exponentiation -> This is causing a bit of problem
return x;
}
}.parse();
}
import java.util.*;
public class check {
int ans;
String str="7 + 5";
StringTokenizer st=new StringTokenizer(str);
int v1=Integer.parseInt(st.nextToken());
String op=st.nextToken();
int v2=Integer.parseInt(st.nextToken());
if(op.equals("+")) { ans= v1 + v2; }
if(op.equals("-")) { ans= v1 - v2; }
//.........
}
I think what ever way you do this it's going to involve a lot of conditional statements. But for single operations like in your examples you could limit it to 4 if statements with something like
String math = "1+4";
if (math.split("+").length == 2) {
//do calculation
} else if (math.split("-").length == 2) {
//do calculation
} ...
It gets a whole lot more complicated when you want to deal with multiple operations like "4+5*6".
If you are trying to build a calculator then I'd surgest passing each section of the calculation separatly (each number or operator) rather than as a single string.
You might have a look at the Symja framework:
ExprEvaluator util = new ExprEvaluator();
IExpr result = util.evaluate("10-40");
System.out.println(result.toString()); // -> "-30"
Take note that definitively more complex expressions can be evaluated:
// D(...) gives the derivative of the function Sin(x)*Cos(x)
IAST function = D(Times(Sin(x), Cos(x)), x);
IExpr result = util.evaluate(function);
// print: Cos(x)^2-Sin(x)^2
package ExpressionCalculator.expressioncalculator;
import java.text.DecimalFormat;
import java.util.Scanner;
public class ExpressionCalculator {
private static String addSpaces(String exp){
//Add space padding to operands.
//https://regex101.com/r/sJ9gM7/73
exp = exp.replaceAll("(?<=[0-9()])[\\/]", " / ");
exp = exp.replaceAll("(?<=[0-9()])[\\^]", " ^ ");
exp = exp.replaceAll("(?<=[0-9()])[\\*]", " * ");
exp = exp.replaceAll("(?<=[0-9()])[+]", " + ");
exp = exp.replaceAll("(?<=[0-9()])[-]", " - ");
//Keep replacing double spaces with single spaces until your string is properly formatted
/*while(exp.indexOf(" ") != -1){
exp = exp.replace(" ", " ");
}*/
exp = exp.replaceAll(" {2,}", " ");
return exp;
}
public static Double evaluate(String expr){
DecimalFormat df = new DecimalFormat("#.####");
//Format the expression properly before performing operations
String expression = addSpaces(expr);
try {
//We will evaluate using rule BDMAS, i.e. brackets, division, power, multiplication, addition and
//subtraction will be processed in following order
int indexClose = expression.indexOf(")");
int indexOpen = -1;
if (indexClose != -1) {
String substring = expression.substring(0, indexClose);
indexOpen = substring.lastIndexOf("(");
substring = substring.substring(indexOpen + 1).trim();
if(indexOpen != -1 && indexClose != -1) {
Double result = evaluate(substring);
expression = expression.substring(0, indexOpen).trim() + " " + result + " " + expression.substring(indexClose + 1).trim();
return evaluate(expression.trim());
}
}
String operation = "";
if(expression.indexOf(" / ") != -1){
operation = "/";
}else if(expression.indexOf(" ^ ") != -1){
operation = "^";
} else if(expression.indexOf(" * ") != -1){
operation = "*";
} else if(expression.indexOf(" + ") != -1){
operation = "+";
} else if(expression.indexOf(" - ") != -1){ //Avoid negative numbers
operation = "-";
} else{
return Double.parseDouble(expression);
}
int index = expression.indexOf(operation);
if(index != -1){
indexOpen = expression.lastIndexOf(" ", index - 2);
indexOpen = (indexOpen == -1)?0:indexOpen;
indexClose = expression.indexOf(" ", index + 2);
indexClose = (indexClose == -1)?expression.length():indexClose;
if(indexOpen != -1 && indexClose != -1) {
Double lhs = Double.parseDouble(expression.substring(indexOpen, index));
Double rhs = Double.parseDouble(expression.substring(index + 2, indexClose));
Double result = null;
switch (operation){
case "/":
//Prevent divide by 0 exception.
if(rhs == 0){
return null;
}
result = lhs / rhs;
break;
case "^":
result = Math.pow(lhs, rhs);
break;
case "*":
result = lhs * rhs;
break;
case "-":
result = lhs - rhs;
break;
case "+":
result = lhs + rhs;
break;
default:
break;
}
if(indexClose == expression.length()){
expression = expression.substring(0, indexOpen) + " " + result + " " + expression.substring(indexClose);
}else{
expression = expression.substring(0, indexOpen) + " " + result + " " + expression.substring(indexClose + 1);
}
return Double.valueOf(df.format(evaluate(expression.trim())));
}
}
}catch(Exception exp){
exp.printStackTrace();
}
return 0.0;
}
public static void main(String args[]){
Scanner scanner = new Scanner(System.in);
System.out.print("Enter an Mathematical Expression to Evaluate: ");
String input = scanner.nextLine();
System.out.println(evaluate(input));
}
}
A Java class that can evaluate mathematical expressions:
package test;
public class Calculator {
public static Double calculate(String expression){
if (expression == null || expression.length() == 0) {
return null;
}
return calc(expression.replace(" ", ""));
}
public static Double calc(String expression) {
String[] containerArr = new String[]{expression};
double leftVal = getNextOperand(containerArr);
expression = containerArr[0];
if (expression.length() == 0) {
return leftVal;
}
char operator = expression.charAt(0);
expression = expression.substring(1);
while (operator == '*' || operator == '/') {
containerArr[0] = expression;
double rightVal = getNextOperand(containerArr);
expression = containerArr[0];
if (operator == '*') {
leftVal = leftVal * rightVal;
} else {
leftVal = leftVal / rightVal;
}
if (expression.length() > 0) {
operator = expression.charAt(0);
expression = expression.substring(1);
} else {
return leftVal;
}
}
if (operator == '+') {
return leftVal + calc(expression);
} else {
return leftVal - calc(expression);
}
}
private static double getNextOperand(String[] exp){
double res;
if (exp[0].startsWith("(")) {
int open = 1;
int i = 1;
while (open != 0) {
if (exp[0].charAt(i) == '(') {
open++;
} else if (exp[0].charAt(i) == ')') {
open--;
}
i++;
}
res = calc(exp[0].substring(1, i - 1));
exp[0] = exp[0].substring(i);
} else {
int i = 1;
if (exp[0].charAt(0) == '-') {
i++;
}
while (exp[0].length() > i && isNumber((int) exp[0].charAt(i))) {
i++;
}
res = Double.parseDouble(exp[0].substring(0, i));
exp[0] = exp[0].substring(i);
}
return res;
}
private static boolean isNumber(int c) {
int zero = (int) '0';
int nine = (int) '9';
return (c >= zero && c <= nine) || c =='.';
}
public static void main(String[] args) {
System.out.println(calculate("(((( -6 )))) * 9 * -1"));
System.out.println(calc("(-5.2+-5*-5*((5/4+2)))"));
}
}
How about something like this:
String st = "10+3";
int result;
for(int i=0;i<st.length();i++)
{
if(st.charAt(i)=='+')
{
result=Integer.parseInt(st.substring(0, i))+Integer.parseInt(st.substring(i+1, st.length()));
System.out.print(result);
}
}
and do the similar thing for every other mathematical operator accordingly ..
It is possible to convert any expression string in infix notation to a postfix notation using Djikstra's shunting-yard algorithm. The result of the algorithm can then serve as input to the postfix algorithm with returns the result of the expression.
I wrote an article about it here, with an implementation in java
Yet another option: https://github.com/stefanhaustein/expressionparser
I have implemented this to have a simple but flexible option to permit both:
Immediate processing (Calculator.java, SetDemo.java)
Building and processing a parse tree (TreeBuilder.java)
The TreeBuilder linked above is part of a CAS demo package that does symbolic derivation. There is also a BASIC interpreter example and I have started to build a TypeScript interpreter using it.
External library like RHINO or NASHORN can be used to run javascript. And javascript can evaluate simple formula without parcing the string. No performance impact as well if code is written well.
Below is an example with RHINO -
public class RhinoApp {
private String simpleAdd = "(12+13+2-2)*2+(12+13+2-2)*2";
public void runJavaScript() {
Context jsCx = Context.enter();
Context.getCurrentContext().setOptimizationLevel(-1);
ScriptableObject scope = jsCx.initStandardObjects();
Object result = jsCx.evaluateString(scope, simpleAdd , "formula", 0, null);
Context.exit();
System.out.println(result);
}
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
public class test2 {
public static void main(String[] args) throws ScriptException {
String s = "10+2";
ScriptEngineManager mn = new ScriptEngineManager();
ScriptEngine en = mn.getEngineByName("js");
Object result = en.eval(s);
System.out.println(result);
}
}
I have done using iterative parsing and shunting Yard algorithm and i have really enjoyed developing the expression evaluator ,you can find all the code here
https://github.com/nagaraj200788/JavaExpressionEvaluator
Has 73 test cases and even works for Bigintegers,Bigdecimals
supports all relational, arithmetic expression and also combination of both .
even supports ternary operator .
Added enhancement to support signed numbers like -100+89 it was intresting, for details check TokenReader.isUnaryOperator() method and i have updated code in above Link

How to make a report parameter selection change another report parameter selection values while allowing multiselect for both of them?

Lets say I have a table with a list of names, such as "a, b, c" and each name has several other values assigned (some of the other values can be assigned to several/all of names values, example below).
Table example:
( names - other ):
a - aa
a - ab
a - ac
b - ab
b - bb
b - cb
c - ac
c - bc
c - cc
How do I make in birt so that I could select names in one parameter box and get corresponding other values to select for another parameter? I know it's possible to do that with cascading parameter, but that doesn't allow to have multiselect for the first parameter, which in our example would be names values.
Found a solution partly here: https://forums.opentext.com/forums/developer/discussion/61283/allow-multi-value-for-cascading-parameter
and here (thanks google translate ^^): https://www.developpez.net/forums/d1402270/logiciels/solutions-d-entreprise/business-intelligence/birt/birt-4-3-1-multi-value-cascade-parameters/
Steps to do:
change"\birt\webcontent\birt\ajax\ui\dialog\BirtParameterDialog.js" file contents (there is a file to download for replacement in one of the links, but in case it goes dead I'm sharing 2 snippets from it where the changes occur:
__refresh_cascade_select : function( element )
{
var matrix = new Array( );
var m = 0;
for( var i = 0; i < this.__cascadingParameter.length; i++ )
{
for( var j = 0; j < this.__cascadingParameter[i].length; j++ )
{
var paramName = this.__cascadingParameter[i][j].name;
if( paramName == this.__isnull )
paramName = this.__cascadingParameter[i][j].value;
if( paramName == element.id.substr( 0, element.id.length - 10 ) )
{
//CHANGE//
//The two way work (String(selectedValue) or Array(SelectedValueArray))
var l = 0;
var isFirstElement = true;
var selectedValue = "";
var selectedValueArray = new Array();
for(var test = 0; test<element.options.length;test++){
if(element.options[test].selected){
if(isFirstElement){
isFirstElement = false;
selectedValue = element.options[test].value;
}
else{
selectedValue = selectedValue+","+element.options[test].value;
}
selectedValueArray[l] = element.options[test].value;
l++;
}
}
//CHANGE//
var tempText = element.options[element.selectedIndex].text;
var tempValue = element.options[element.selectedIndex].value;
// Null Value Parameter
if ( tempValue == Constants.nullValue )
{
this.__cascadingParameter[i][j].name = this.__isnull;
this.__cascadingParameter[i][j].value = paramName;
}
else if( tempValue == '' )
{
if( tempText == "" )
{
var target = element;
target = target.parentNode;
var oInputs = target.getElementsByTagName( "input" );
if( oInputs.length >0 && oInputs[1].value != Constants.TYPE_STRING )
{
// Only String parameter allows blank value
alert( birtUtility.formatMessage( Constants.error.parameterNotAllowBlank, paramName ) );
this.__clearSubCascadingParameter( this.__cascadingParameter[i], j );
return;
}
else
{
// Blank Value
this.__cascadingParameter[i][j].name = paramName;
this.__cascadingParameter[i][j].value = tempValue;
}
}
else
{
// Blank Value
this.__cascadingParameter[i][j].name = paramName;
this.__cascadingParameter[i][j].value = tempValue;
}
}
else
{
this.__cascadingParameter[i][j].name = paramName;
//CHANGE//
//The two way work (String(selectedValue) or Array(SelectedValueArray))
this.__cascadingParameter[i][j].value = selectedValueArray;
//CHANGE//
}
for( var m = 0; m <= j; m++ )
{
if( !matrix[m] )
{
matrix[m] = {};
}
matrix[m].name = this.__cascadingParameter[i][m].name;
matrix[m].value = this.__cascadingParameter[i][m].value;
}
this.__pendingCascadingCalls++;
birtEventDispatcher.broadcastEvent( birtEvent.__E_CASCADING_PARAMETER, matrix );
}
}
}
},
and this:
// exist select control and input text/password
// compare the parent div offsetTop
if( oFirstITC.parentNode && oFirstST.parentNode )
{
// Bugzilla 265615: need to use cumulative offset for special cases
// where one element is inside a group container
var offsetITC = Position.cumulativeOffset( oFirstITC );
var offsetST = Position.cumulativeOffset( oFirstST );
// compare y-offset first, then x-offset to determine the visual order
if( ( offsetITC[1] > offsetST[1] ) || ( offsetITC[1] == offsetST[1] && offsetITC[0] > offsetST[0] ) )
{
oFirstST.focus( );
}
else
{
oFirstITC.focus( );
}
}
After .js is changed cascading parameters can have multiselect on all levels.
Example:
1st DataSet "DS_country" query:
select CLASSICMODELS.OFFICES.COUNTRY
from CLASSICMODELS.OFFICES
2nd DataSet "DS_office" query:
select CLASSICMODELS.OFFICES.OFFICECODE
from CLASSICMODELS.OFFICES
where CLASSICMODELS.OFFICES.COUNTRY IN ('DS_country')
After datasets are created we can make cascading report parameters CRP_country and CRP_office (keep in mind that UI doesn't let you to choose "Allow multiple values" on the upper levels, but we can change that in property editor after the parameters are made by going to advanced tab and changing "Scalar parameter type" property value to "Multi Value")
The only thing left is to pick lower level cascading parameters (CRP_office in our example) and go to script tab and add this on "beforeOpen":
// Do that if your selections are in String type.
var stringArray = params["CRP_country"].value.toString().split(",");
var result = "";
for(var i =0 ; i < stringArray.length ; i++){
if(i==0){
result = "'"+stringArray[i]+"'";
}
else{
result = result+",'"+stringArray[i]+"'";
}
}
// For String
this.queryText = this.queryText.replace("'DS_country'", result);
//For integer (the first part is useless)
//this.queryText = this.queryText.replace("'DS_country'",
params["CRP_country"].value);

Generate N-grams while preserving spaces in apache lucene

I am trying to generate N-grams using apache Lucene 5.5.4 for a given set input text. Following is my java code to do the same.
public static void main( String[] args )
{
Analyzer analyzer = createAnalyzer( 2 );
List<String> nGrams = generateNgrams( analyzer, "blah1 blah2 blah3" );
for ( String nGram : nGrams ) {
System.out.println( nGram );
}
}
public static Analyzer createAnalyzer( final int shingles )
{
return new Analyzer() {
#Override
protected TokenStreamComponents createComponents( #NotNull String field )
{
final Tokenizer source = new WhitespaceTokenizer();
final ShingleFilter shingleFilter = new ShingleFilter( new LowerCaseFilter( source ), shingles );
shingleFilter.setOutputUnigrams( true );
return new TokenStreamComponents( source, shingleFilter );
}
};
}
public static List<String> generateNgrams( Analyzer analyzer, String str )
{
List<String> result = new ArrayList<>();
try {
TokenStream stream = analyzer.tokenStream( null, new StringReader( str ) );
stream.reset();
while ( stream.incrementToken() ) {
String nGram = stream.getAttribute( CharTermAttribute.class ).toString();
result.add( nGram );
LOG.debug( "Generated N-gram = {}", nGram );
}
} catch ( IOException e ) {
LOG.error( "IO Exception occured! {}", e );
}
return result;
}
For my input blah1 blah2 blah3, the output is as follows and i am okay with it.
blah1
blah1 blah2
blah2
blah2 blah3
blah3
However, when the input is Foo bar Foo2, my requirement is to generate the following output:
Foo
Foo bar
bar
bar Foo2
Foo2
If you noticed, I have to preserve the spaces in between 2 words as it is in the input.(Foo bar and not Foo bar).
Can I make any tweaks and ask lucene to handle it internally?
May be its a minor tweak like adding a filter or something and since I am new to Lucene, I don't know where to start.
Thanks in Advance.
I had to write custom tokenizers and and trim filters to achieve this.
1) I created an abstract class DelimiterPreservingCharTokenizer by extending org.apache.lucene.analysis.Tokenizer class. Next, gave my implementation for incrementToken method. I would have extended org.apache.lucene.analysis.util.CharTokenizer if not the class was final. DelimiterPreservingCharTokenizer looks like below.
package lucene.tokenizers;
import java.io.IOException;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
import org.apache.lucene.analysis.util.CharTokenizer;
import org.apache.lucene.analysis.util.CharacterUtils;
import org.apache.lucene.analysis.util.CharacterUtils.CharacterBuffer;
import org.apache.lucene.util.AttributeFactory;
/**
*
* #author Arun Gowda.
* This class is exactly same as {#link CharTokenizer}. Except that, the stream will have leading delimiters. This is to support N-gram vicinity matches.
*
* We are creating a new class instead of extending CharTokenizer because, incrementToken method is final and we can not override it.
*
*/
public abstract class DelimiterPreservingCharTokenizer extends Tokenizer
{
/**
* Creates a new {#link DelimiterPreservingCharTokenizer} instance
*/
public DelimiterPreservingCharTokenizer()
{}
/**
* Creates a new {#link DelimiterPreservingCharTokenizer} instance
*
* #param factory
* the attribute factory to use for this {#link Tokenizer}
*/
public DelimiterPreservingCharTokenizer( AttributeFactory factory )
{
super( factory );
}
private int offset = 0, bufferIndex = 0, dataLen = 0, finalOffset = 0;
private static final int MAX_WORD_LEN = 255;
private static final int IO_BUFFER_SIZE = 4096;
private final CharTermAttribute termAtt = addAttribute( CharTermAttribute.class );
private final OffsetAttribute offsetAtt = addAttribute( OffsetAttribute.class );
private final CharacterUtils charUtils = CharacterUtils.getInstance();
private final CharacterBuffer ioBuffer = CharacterUtils.newCharacterBuffer( IO_BUFFER_SIZE );
/**
* Returns true iff a codepoint should be included in a token. This tokenizer
* generates as tokens adjacent sequences of codepoints which satisfy this
* predicate. Codepoints for which this is false are used to define token
* boundaries and are not included in tokens.
*/
protected abstract boolean isTokenChar( int c );
/**
* Called on each token character to normalize it before it is added to the
* token. The default implementation does nothing. Subclasses may use this to,
* e.g., lowercase tokens.
*/
protected int normalize( int c )
{
return c;
}
#Override
public final boolean incrementToken() throws IOException
{
clearAttributes();
int length = 0;
int start = -1; // this variable is always initialized
int end = -1;
char[] buffer = termAtt.buffer();
while ( true ) {
if ( bufferIndex >= dataLen ) {
offset += dataLen;
charUtils.fill( ioBuffer, input ); // read supplementary char aware with CharacterUtils
if ( ioBuffer.getLength() == 0 ) {
dataLen = 0; // so next offset += dataLen won't decrement offset
if ( length > 0 ) {
break;
} else {
finalOffset = correctOffset( offset );
return false;
}
}
dataLen = ioBuffer.getLength();
bufferIndex = 0;
}
// use CharacterUtils here to support < 3.1 UTF-16 code unit behavior if the char based methods are gone
final int c = charUtils.codePointAt( ioBuffer.getBuffer(), bufferIndex, ioBuffer.getLength() );
final int charCount = Character.charCount( c );
bufferIndex += charCount;
if ( isTokenChar( c ) ) { // if it's a token char
if ( length == 0 ) { // start of token
assert start == -1;
start = offset + bufferIndex - charCount;
end = start;
} else if ( length >= buffer.length - 1 ) { // check if a supplementary could run out of bounds
buffer = termAtt.resizeBuffer( 2 + length ); // make sure a supplementary fits in the buffer
}
end += charCount;
length += Character.toChars( normalize( c ), buffer, length ); // buffer it, normalized
if ( length >= MAX_WORD_LEN ) // buffer overflow! make sure to check for >= surrogate pair could break == test
break;
} else if ( length > 0 ) // at non-Letter w/ chars
break; // return 'em
}
if ( length > 0 && bufferIndex < ioBuffer.getLength() ) {//If at least one token is found,
//THIS IS THE PART WHICH IS DIFFERENT FROM LUCENE's CHARTOKENIZER
// use CharacterUtils here to support < 3.1 UTF-16 code unit behavior if the char based methods are gone
int c = charUtils.codePointAt( ioBuffer.getBuffer(), bufferIndex, ioBuffer.getLength() );
int charCount = Character.charCount( c );
bufferIndex += charCount;
while ( !isTokenChar( c ) && bufferIndex < ioBuffer.getLength() ) {// As long as we find delimiter(not token char), keep appending it to output stream.
if ( length >= buffer.length - 1 ) { // check if a supplementary could run out of bounds
buffer = termAtt.resizeBuffer( 2 + length ); // make sure a supplementary fits in the buffer
}
end += charCount;
length += Character.toChars( normalize( c ), buffer, length ); // buffer it, normalized
if ( length >= MAX_WORD_LEN ) {// buffer overflow! make sure to check for >= surrogate pair could break == test
break;
}
c = charUtils.codePointAt( ioBuffer.getBuffer(), bufferIndex, ioBuffer.getLength() );
charCount = Character.charCount( c );
bufferIndex += charCount;
}
//ShingleFilter will add a delimiter. Hence, the last iteration is skipped.
//That is, for input `abc def ghi`, this tokenizer will return `abc `(2 spaces only). Then, Shingle filter will by default add another delimiter making it `abc `(3 spaces as it is in the input).
//If there are N delimiters, this token will at max return N-1 delimiters
bufferIndex -= charCount;
}
termAtt.setLength( length );
assert start != -1;
offsetAtt.setOffset( correctOffset( start ), finalOffset = correctOffset( end ) );
return true;
}
#Override
public final void end() throws IOException
{
super.end();
// set final offset
offsetAtt.setOffset( finalOffset, finalOffset );
}
#Override
public void reset() throws IOException
{
super.reset();
bufferIndex = 0;
offset = 0;
dataLen = 0;
finalOffset = 0;
ioBuffer.reset(); // make sure to reset the IO buffer!!
}
}
2) A concrete class WhiteSpacePreservingTokenizer extending the above abstract class to provide delimiter
package spellcheck.lucene.tokenizers;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.core.WhitespaceTokenizer;
import org.apache.lucene.util.AttributeFactory;
/**
*
* #author Arun Gowda
*
* This class is exactly same as {#link WhitespaceTokenizer} Only difference is, it extends DelimiterPreservingCharTokenizer instead of CharTokenizer
*/
public class WhiteSpacePreservingTokenizer extends DelimiterPreservingCharTokenizer
{
/**
* Construct a new WhitespaceTokenizer.
*/
public WhiteSpacePreservingTokenizer()
{}
/**
* Construct a new WhitespaceTokenizer using a given
* {#link org.apache.lucene.util.AttributeFactory}.
*
* #param factory
* the attribute factory to use for this {#link Tokenizer}
*/
public WhiteSpacePreservingTokenizer( AttributeFactory factory )
{
super( factory );
}
/** Collects only characters which do not satisfy
* {#link Character#isWhitespace(int)}.*/
#Override
protected boolean isTokenChar( int c )
{
return !Character.isWhitespace( c );
}
}
3) The tokenizer above will result in tailing spaces. (Ex: blah____) we need to add a filter to trim those spaces. So we need DelimiterTrimFilter as folows.(We can also just trim by using java's trim. but doing so will be very inefficient since it creates new string)
package spellcheck.lucene.filters;
import java.io.IOException;
import org.apache.lucene.analysis.TokenFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
public class DelimiterTrimFilter extends TokenFilter
{
private final CharTermAttribute termAtt = addAttribute( CharTermAttribute.class );
private char delimiter;
/**
* Create a new {#link DelimiterTrimFilter}.
* #param in the stream to consume
* #param delimiterToTrim delimiter that should be trimmed
*/
public DelimiterTrimFilter( TokenStream in, char delimiterToTrim )
{
super( in );
this.delimiter = delimiterToTrim;
}
#Override
public boolean incrementToken() throws IOException
{
if ( !input.incrementToken() )
return false;
char[] termBuffer = termAtt.buffer();
int len = termAtt.length();
if ( len == 0 ) {
return true;
}
int start = 0;
int end = 0;
// eat the first characters
for ( start = 0; start < len && termBuffer[start] == delimiter; start++ ) {
}
// eat the end characters
for ( end = len; end >= start && termBuffer[end - 1] == delimiter; end-- ) {
}
if ( start > 0 || end < len ) {
if ( start < end ) {
termAtt.copyBuffer( termBuffer, start, ( end - start ) );
} else {
termAtt.setEmpty();
}
}
return true;
}
}
4) My createAnalyzer will look like below
public static Analyzer createAnalyzer( final int shingles )
{
return new Analyzer() {
#Override
protected TokenStreamComponents createComponents( #NotNull String field )
{
final Tokenizer source = new WhiteSpacePreservingTokenizer();
final TokenStream filter = new ShingleFilter( new LowerCaseFilter( source ), shingles );
filter = new DelimiterTrimFilter( filter, ' ' );
return new TokenStreamComponents( source, filter );
}
};
}
Rest of the code will remain the same

Revit Transform Matrix from point cloud to revit link

We insert a point cloud in Revit and have to move and rotate it in several axles to get a good world alignment. We then need to apply the same transforms to a Revit MEP link that contains edgwise piping.
I have acquired the 3x4 matrix from the point cloud in revit, I would like to apply that to the revit mep link so that they are in alignment.
I have successfully applied the translation portion of the transform, I am stuck on the rotation portion. Could someone please assist with this. Dyslexia and Matrices do not work well together.
Thanks to Jeremy at the Building Coder for getting me this far!
9/4/19 - I edited the rotational part of the code, however the "ElementTransformUtils.RotateElement" does not have any effect on the link in Revit.
Autodesk.Revit.DB.View
pView = ActiveUIDocument.Document.ActiveView;Autodesk.Revit.DB.Transaction
t = new Autodesk.Revit.DB.Transaction(ActiveUIDocument.Document, "EdgewiseCoordination");
t.Start();
UIDocument uidoc = this.ActiveUIDocument;
Document document = uidoc.Document;
Autodesk.Revit.ApplicationServices.Application application = document.Application;
ElementId elementid = null;
elementid = uidoc.Selection.PickObject(ObjectType.Element, "Select element to Copy Transform or ESC to reset the view").ElementId;
Element e = document.GetElement(elementid);
Instance ei = e as Instance;
Transform pct = ei.GetTransform();
ElementId elementid2 = null;
elementid2 = uidoc.Selection.PickObject(ObjectType.Element, "Select element to Apply Transform or ESC to reset the view").ElementId;
Element e2 = document.GetElement(elementid2);
Instance ei2 = e2 as Instance;
ElementTransformUtils.MoveElement(document,elementid2,pct.Origin);
Line lineaxis = GetRotationAxisFromTransform(pct);
double tangle = GetRotationAngleFromTransform(pct);
ElementTransformUtils.RotateElement(document,elementid2,lineaxis,tangle);
t.Commit();
}
private static Line GetRotationAxisFromTransform(Transform transform)
{
double x = transform.BasisY.Z - transform.BasisZ.Y;
double y = transform.BasisZ.X - transform.BasisX.Z;
double z = transform.BasisX.Y - transform.BasisY.X;
return Line.CreateUnbound(transform.Origin, new XYZ(x, y, z));
}
private static double GetRotationAngleFromTransform(Transform transform)
{
double x = transform.BasisX.X;
double y = transform.BasisY.Y;
double z = transform.BasisZ.Z;
double trace = x + y + z;
return Math.Acos((trace - 1) / 2.0);
}
Earlier Code to save out the transform----------------------------------
public void Objectlocation()
{
Autodesk.Revit.DB.View
pView = ActiveUIDocument.Document.ActiveView;Autodesk.Revit.DB.Transaction
t = new Autodesk.Revit.DB.Transaction(ActiveUIDocument.Document, "Objectlocation");
t.Start();
UIDocument uidoc = this.ActiveUIDocument;
Document document = uidoc.Document;
Autodesk.Revit.ApplicationServices.Application application = document.Application;
ElementId elementid = null;
elementid = uidoc.Selection.PickObject(ObjectType.Element, "Select element or ESC to reset the view").ElementId;
Element e = document.GetElement(elementid);
Instance ei = e as Instance;
Transform pct = ei.GetTransform();
//string spctrans = TransformString(pct);
//TaskDialog.Show("Point Cloud Transform", spctrans);
//spctrans = (spctrans + ",0,0,0,1");
string slocx = TransformStringX(pct);
string slocy = TransformStringY(pct);
string slocz = TransformStringZ(pct);
string slocation = (slocx + "," + slocy + "," + slocz + ",0,0,0,1");
using (StreamWriter sw = new StreamWriter("test.txt"))
{
sw.WriteLine(slocation);
}
t.Commit();
}
static public string RealString( double a )
{
return a.ToString( "0.####################" );
}
static public string PointStringX( XYZ p )
{
double convunit = 0;
convunit = UnitUtils.ConvertFromInternalUnits(p.X,DisplayUnitType.DUT_DECIMAL_INCHES);
return string.Format( "{0}", RealString( convunit ));
}
static public string TransformStringX( Transform t )
{
return string.Format( "{0},{1},{2},{3}", PointStringX( t.BasisX ),
PointStringX( t.BasisY ), PointStringX( t.BasisZ ), PointStringX( t.Origin ) );
}
static public string PointStringY( XYZ p )
{
double convunit = 0;
convunit = UnitUtils.ConvertFromInternalUnits(p.Y,DisplayUnitType.DUT_DECIMAL_INCHES);
return string.Format( "{0}", RealString( convunit ));
}
static public string TransformStringY( Transform t )
{
return string.Format( "{0},{1},{2},{3}", PointStringY( t.BasisX ),
PointStringY( t.BasisY ), PointStringY( t.BasisZ ), PointStringY( t.Origin ) );
}
static public string PointStringZ( XYZ p )
{
double convunit = 0;
convunit = UnitUtils.ConvertFromInternalUnits(p.Z,DisplayUnitType.DUT_DECIMAL_INCHES);
return string.Format( "{0}", RealString( convunit ));
}
static public string TransformStringZ( Transform t )
{
return string.Format( "{0},{1},{2},{3}", PointStringZ( t.BasisX ),
PointStringZ( t.BasisY ), PointStringZ( t.BasisZ ), PointStringZ( t.Origin ) );
}
I edited the stuff you had on bcoder to split up the matrix in order to match the acad api that I was feeding it into.
ACAD API below-----------------------------------------------------------
using Autodesk.AutoCAD.ApplicationServices;
using Autodesk.AutoCAD.EditorInput;
using Autodesk.AutoCAD.DatabaseServices;
using Autodesk.AutoCAD.Geometry;
using Autodesk.AutoCAD.Runtime;
using System.Reflection;
namespace Transformer
{
public class Commands
{
[CommandMethod("TRANS", CommandFlags.UsePickSet)]
static public void TransformEntity()
{
Document doc =
Application.DocumentManager.MdiActiveDocument;
Database db = doc.Database;
Editor ed = doc.Editor;
// Our selected entity (only one supported, for now)
ObjectId id;
// First query the pickfirst selection set
PromptSelectionResult psr = ed.SelectImplied();
if (psr.Status != PromptStatus.OK || psr.Value == null)
{
// If nothing selected, ask the user
PromptEntityOptions peo =
new PromptEntityOptions(
"\nSelect entity to transform: "
);
PromptEntityResult per = ed.GetEntity(peo);
if (per.Status != PromptStatus.OK)
return;
id = per.ObjectId;
}
else
{
// If the pickfirst set has one entry, take it
SelectionSet ss = psr.Value;
if (ss.Count != 1)
{
ed.WriteMessage(
"\nThis command works on a single entity."
);
return;
}
ObjectId[] ids = ss.GetObjectIds();
id = ids[0];
}
PromptResult pr = ed.GetString("\nEnter property name: ");
if (pr.Status != PromptStatus.OK)
return;
string prop = pr.StringResult;
// Now let's ask for the matrix string
pr = ed.GetString("\nEnter matrix values: ");
if (pr.Status != PromptStatus.OK)
return;
// Split the string into its individual cells
string[] cells = pr.StringResult.Split(new char[] { ',' });
if (cells.Length != 16)
{
ed.WriteMessage("\nMust contain 16 entries.");
return;
}
try
{
// Convert the array of strings into one of doubles
double[] data = new double[cells.Length];
for (int i = 0; i < cells.Length; i++)
{
data[i] = double.Parse(cells[i]);
}
// Create a 3D matrix from our cell data
Matrix3d mat = new Matrix3d(data);
// Now we can transform the selected entity
Transaction tr =
doc.TransactionManager.StartTransaction();
using (tr)
{
Entity ent =
tr.GetObject(id, OpenMode.ForWrite)
as Entity;
if (ent != null)
{
bool transformed = false;
// If the user specified a property to modify
if (!string.IsNullOrEmpty(prop))
{
// Query the property's value
object val =
ent.GetType().InvokeMember(
prop, BindingFlags.GetProperty, null, ent, null
);
// We only know how to transform points and vectors
if (val is Point3d)
{
// Cast and transform the point result
Point3d pt = (Point3d)val,
res = pt.TransformBy(mat);
// Set it back on the selected object
ent.GetType().InvokeMember(
prop, BindingFlags.SetProperty, null,
ent, new object[] { res }
);
transformed = true;
}
else if (val is Vector3d)
{
// Cast and transform the vector result
Vector3d vec = (Vector3d)val,
res = vec.TransformBy(mat);
// Set it back on the selected object
ent.GetType().InvokeMember(
prop, BindingFlags.SetProperty, null,
ent, new object[] { res }
);
transformed = true;
}
}
// If we didn't transform a property,
// do the whole object
if (!transformed)
ent.TransformBy(mat);
}
tr.Commit();
}
}
catch (Autodesk.AutoCAD.Runtime.Exception ex)
{
ed.WriteMessage(
"\nCould not transform entity: {0}", ex.Message
);
}
}
}
}
END ACAD API

JDBC if a string exist in a database

I am trying to find where a specific string exist in a database (all tables). I have the following code:
DatabaseMetaData md = con.getMetaData();
ResultSet rs = md.getTables(null, null, "%", null);
while (rs.next()) {
stm = con.createStatement();
String sql;
sql = "SELECT * FROM "+rs.getString(3)+"WHERE F01 = '0000000000998'";
rs2 = stm.executeQuery(sql);
while(rs2.next()){
System.out.println(rs.getString(3));
}
}
The problem is in some of the tables F01 doesn't exist, so it throws an exception. Is there any way that even without specifying the column I can search through the whole table?
which database you are using.
Look over this discussion post. you may get good ideas about how to make sure column exists before you call your check
http://www.coderanch.com/t/299298/JDBC/databases/Oracle-describe-table-jdbc
For oracle
you can check these queries
select COLUMN_NAME, DATA_LENGTH, DATA_TYPE from user_tab_columns where Lower(table_name) = 'product'
select table_name, COLUMN_NAME, DATA_LENGTH, DATA_TYPE from user_tab_columns where upper(column_name) = 'PRODUCTID'
therefore your call should be something like this
select table_name from user_tab_columns where upper(column_name) = 'F01'
and then
SELECT * FROM "+rs.getString(1)+"WHERE F01 = '0000000000998'
as you see I am using LOWER and UPPER, you need to make sure you include them, reason as you can understand even though SQL is not case sensitive in its statements, but the value for which conditional check is happening is case sensitive.
The following Java code loops through each VARCHAR/NVARCHAR column in each table and performs a SELECT TOP 1 on that column to see if it gets a hit:
import java.sql.*;
import java.util.ArrayList;
import java.util.List;
public class JDBCQuery {
public static void main(String args[]) {
String textToSearchFor = "Gord"; // test data
System.out.println(String.format("The value '%s' was found in the following locations:", textToSearchFor));
Connection conn = null;
PreparedStatement ps = null;
try {
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
String connectionString =
"jdbc:odbc:Driver={SQL Server};" +
"Server=.\\SQLEXPRESS;" +
"Trusted_Connection=yes;" +
"Database=myDb";
conn = DriverManager.getConnection(connectionString);
DatabaseMetaData md = conn.getMetaData();
ResultSet mdrs = md.getTables(null, null, "%", new String[] { "TABLE" });
List<String> tableList = new ArrayList<String>();
while (mdrs.next()) {
tableList.add(String.format("[%s].[%s].[%s]", mdrs.getString(1), mdrs.getString(2), mdrs.getString(3)));
// i.e., [catalogName].[schemaName].[tableName]
}
mdrs.close();
for (String tableName : tableList) {
Statement stmt = conn.createStatement();
stmt.execute("SELECT * FROM " + tableName + " WHERE 0=1");
ResultSet rs0 = stmt.getResultSet();
ResultSetMetaData rsmd = rs0.getMetaData();
List<String> columnList = new ArrayList<String>();
for (int colIndex = 1; colIndex <= rsmd.getColumnCount(); colIndex++) {
switch (rsmd.getColumnType(colIndex)) {
case java.sql.Types.VARCHAR:
case java.sql.Types.NVARCHAR:
columnList.add("[" + rsmd.getColumnName(colIndex) + "]");
break;
}
}
rs0.close();
stmt.close();
for (String columnName : columnList) {
String psSql = String.format("SELECT TOP 1 * FROM %s WHERE %s = ?", tableName, columnName);
ps = conn.prepareStatement(psSql);
ps.setString(1, textToSearchFor);
ResultSet rs1 = ps.executeQuery();
if (rs1.next()) {
System.out.println(String.format("column %s in %s", columnName, tableName));
}
rs1.close();
ps.close();
}
}
} catch( Exception e ) {
e.printStackTrace();
} finally {
try {
if (ps != null) {
ps.close();
}
if (conn != null) {
conn.close();
}
} catch( Exception e ) {
e.printStackTrace();
}
}
}
}
The results look like this;
The value 'Gord' was found in the following locations:
column [textCol] in [myDb].[dbo].[linkedTable]
column [FirstName] in [myDb].[dbo].[myContacts]