I have connected my Zeppelin to postgresql using the native driver provided in the Zeppelin package. I have a column - ID (of a table say 'a') which consist of all the ID's required for further processing.
Example
====================================
ID | Value 1 | Value2
====================================
1 | 23 | text1
2 | 13 | text2
3 | 03 | text3
4 | 99 | text4
5 | 12 | text5
Now, How do I fill this ID column into dynamic form for further user drop down selection?
I know I can use spark context and JDBC drive. But that would be a overkill. All my further processing are with the DB itself for analysis. So I would prefer to create dropdown dynamic form from the psql interpreter itself.
Related
I am facing a large table with data that got imported from a csv. However the delimiters in the csv where not sanitized, so the input data looked something like this:
alex#mail.com:Alex
dummy#mail.com;Bob
foo#bar.com:Foo
spam#yahoo.com;Spam
whatever#mail.com:Whatever
During the import : was defined as the delimiter, so each row with the delimiter ; was not imported properly. This resulted in a table structured like this:
| ID | MAIL | USER |
|-- --|---------------------|----------|
| 1 | alex#mail.com | ALEX |
| 2 | dummy#mail.com;Bob | NULL |
| 3 | foo#bar.com | Foo |
| 4 | spam#yahoo.com;Spam | NULL |
| 5 | whatever#mail.com | Whatever |
As reimporting is no option I was thinking about manually sanitizing the data in the affected rows by using SQL queries. So I tried to combine SELECT and UPDATE statements by filtering rows WHERE USER IS NULL and update both columns with the correct value where applicable.
What you need are string functions. Reading a bit, I find that Google BigQuery has STRPOS() and SUBSTR().
https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#substr
https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#strpos
An update query to fix the situation you are describing looks like this:
update table_name set mail =SUBSTR(mail,1,STRPOS(mail,';')-1), user =SUBSTR(mail,STRPOS(mail,';')+1) where user is null
The idea here is to split mail in its two parts, the part before the ; and the part after. Hope this helps.
What is the most efficient way to store a variable amount of columns in MS-SQL?
I have a requirement to store a large number (several million) records into a Microsoft SQL server (via c#). Most columns are standard, but certain groups of users will need to add their own custom columns, and record data in them.
The data in each custom column field will not be large, but the number of records with a certain set of custom columns will be in the millions.
I do not know ahead of time what these columns might be (in terms of name or datatype), but I'll need to pull reports based on these columns as effeciently as possible..
What is the most efficient way of storing the new varying columns and data?
Entity-Attribute-Value model?
Con's: Efficiency if there's a large number of custom columns (= large number of rows)?
A extra table "CustomColumns"?
Storing columnName, Data, Datatype each time an entry has a custom column, for each column.
Con's: A table with a large number of records, perhaps not the most efficient storage.
Serialise the extra columns for each record into a single field
Con's: Lookup efficiency and stored procedure complicated when running reports based on a custom field.
Any other?
Edit: Think I may be confusing option (1) and (2): I actually meant, is the following the best approach :
Entity (User Groups)
id | name | description
-- | ---- | ------------
1 | user group 1 | user group 1
2 | user group 2 | user group 2
Attribute
id | name | type | entityids (best way to do this for 2 user
-- | ---- | ---- | groups using same attribute?
1 | att1 | string | 1,2
2 | att2 | int | 2
3 | att3 | string | 1
4 | att4 | numeric | 2
5 | att5 | string | 1
Value
id | entityId| attributeId | value
-- | --------| ----------- | -----
1 | 1 | 1 | a
2 | 1 | 2 | 1
3 | 1 | 3 | b
4 | 1 | 3 | c
5 | 1 | 3 | d
6 | 1 | 3 | 75
7 | 1 | 5 | Inches
I have a column call it id:
1
1.1
1.2
1.2.1
1.2.2
1.19.1.1
1.2.3.1
1.2.3.2
1.19.1
1.19.1.2
...
etc...
What I'd like to do is include an ORDER BY statement that splits the string, to become like this:
1
1.1
1.2
1.2.1
1.2.2
1.2.3.1
1.2.3.2
1.19.1
1.19.1.1
1.19.1.2
...
etc....
How would I do something like this?
I tried this solution Order By Split Column but doesn't work when length of number are doesn't same
I tried this solution Order By Split Column but doesn't work when length of number are doesn't same
You can make them contain the same number of "." or tokens. For example, if you know that there can be a maximum of 4 dots (eg. 1.1.1.1.1) then you can run this script to concatenate the remaining ".0" tokens :
create table mytable(id varchar);
insert into mytable(id)
values ('1'), ('1.1'), ('1.2'), ('1.2.1'), ('1.2.2'), ('1.19.1.1'), ('1.2.3.1'), ('1.2.3.2'), ('1.19.1'), ('1.19.1.2');
select id as original, id||
case when length(id) - length(replace(id,'.','')) = 0 then ".0.0.0.0"
when length(id) - length(replace(id,'.','')) = 1 then ".0.0.0"
when length(id) - length(replace(id,'.','')) = 2 then ".0.0"
when length(id) - length(replace(id,'.','')) = 3 then ".0"
end as computed
from mytable;
You can run (each sql command at a time) the script here to test it
Result :
----------------------------
| original | computed |
----------------------------
| 1 | 1.0.0.0.0 |
| 1.1 | 1.1.0.0.0 |
| 1.2 | 1.2.0.0.0 |
| 1.2.1 | 1.2.1.0.0 |
| 1.2.2 | 1.2.2.0.0 |
| 1.19.1.1 | 1.19.1.1.0 |
| 1.2.3.1 | 1.2.3.1.0 |
| 1.2.3.2 | 1.2.3.2.0 |
| 1.19.1 | 1.19.1.0.0 |
| 1.19.1.2 | 1.19.1.2.0 |
----------------------------
After this conversion you can apply the script that you mentioned.
As I understand, you need solution for MS Access SQL, not MS SQL. If so, then Gustav's solution from topic you mentioned works fine without any change
I have the following data:
| ID | TYPE | USER_ID |
|----------|----------|----------|
| 1 | A | 7 |
| 1 | A | 8 |
| 1 | B | 6 |
| 2 | A | 9 |
| 2 | B | 5 |
I'm trying to create a query to return
| ID | RESULT |
|----------|----------|
| 1 | 7, 8, 6 |
| 2 | 9, 5 |
The USER_ID values must be ordered by the TYPE attribute.
Since I'm using MS ACCESS, I'm trying to pivot. What I've tried:
TRANSFORM first(user_id)
SELECT id, type
FROM mytable
GROUP BY id, type
ORDER BY type
PIVOT user_id
Error:
Too many crosstab column headers (4547).
I'm missing something in the syntax. However, it seems to be wrong since the first() aggregate needs to be changed to something else to concatenate the results.
PS: I'm using MS-ACCESS 2007. If you know a solution for SQL-Server or Oracle using only SQL (without vendor functions or stored procedures), I'll probably accept your answer since it will help me to find a solution for this problem.
You don't want to use PIVOT. Pivot will create a column named after each of your user IDs (1 - 7). Your TYPE field doesn't seem to do anything either.
Unfortunately, doing this in SQL Server requires the use of a function (FOR XML Path) that's not available in Access.
Here's a link with a similar Access function to do something similar.
In the following feature file my requirement is to put one doc_id for every three field. To clarify, I want to check ProductName, manufacturerName and RevisionDate for every doc_id. I came up with following method but I think this is definitely not the preferred one. Can anyone suggest me a better way.
Background:
Given I am in landigpage page after login
Scenario Outline: valid
When I enter "<doc_id>"
And I click the search go button
Then I should get in vault search page
And Search result of "<field>" should match with database
Examples:
| doc_id | field |
| 15 | ProductName |
| 15 | ManufacturerName |
| 15 | RevisionDate |
Examples:
| doc_id | field |
| 16 | ProductName |
| 16 | ManufacturerName |
| 16 | RevisionDate |
You can use single examples table:
Background:
Given I am in landigpage page after login
Scenario Outline: valid
When I enter "<doc_id>"
And I click the search go button
Then I should get in vault search page
And Search result of "<field>" should match with database
Examples:
| doc_id | field |
| 15 | ProductName |
| 15 | ManufacturerName |
| 15 | RevisionDate |
| 16 | ProductName |
| 16 | ManufacturerName |
| 16 | RevisionDate |
I don't see any other way to pass arguments the way you need to. That's what I don't like about Cucumber. It is not that flexible.
Bacckground:
Given I am in landigpage page after login
Scenario Outline: valid
When I enter "<doc_id>"
And I click the search go button
Then I should get in vault search page
And the search results should match the database
Examples:
| doc_id |
| 15 |
| 16 |
and to make this work:
When /^I enter "<\w+>"$/ do | doc_id |
#doc_id = doc_id
...
end
Then "the search results should match the database" do
db_results = db.find(#doc_id) # or something similar
... # compare db_results to actual results
end
This still kind of sucks, because you have doc_id's in your Gherkin, you are relying on a prefilled database and you have a scenario outline; but hey lets save that for other questions :)
Hi i guess you guys are confusing Data Tables and scenario outline.
The solution for above is :
Background:
Given I am in landingpage page after login
Scenario Outline: valid
When I enter "<doc_id>"
And I select to navigate to search page
Then Search result of field should match with database
| ProductName |
| ManufacturerName |
| RevisionDate |
Examples:
| doc_id |
| 15 |
| 16 |
The table can easily be converted to a list or a map that you can use in your step.