I was asked to create an automation tool to create a report excel sheet using the data from an internal web page of the company I work for. I selected python(3.7) as my language(since most of my coworkers use it as well)and Selenium to do the job. Once I have clicked and filled all the fields using Selenium to filter the data I need, I got at least 60 pages of 20 rows each one(the number can vary), so I have to copy each page data in an excel sheet and click the 'next' button, copy the data again and so on, until all pages are processed. I was able to copy the data for the first page of the web site, but when I try to do the 'next' process, it fails in saving the data in the excel sheet. Here is the code I'm working with
***Bunch of Selenium steps***
def excelFactory(values, row, col): #fuction to create the excel workbook
fname = 'App_Report.xlsx'
if os.path.exists(fname): #check if the workbook is already created
wb = openpyxl.load_workbook(fname)
ws = wb.get_sheet_by_name('Sheet')
else: #create the workbook if not found
wb = Workbook()
ws = wb.active
ws.cell(row= row, column= col).value = values #insert the data from the online web in the
#workbook
row = row + 1
wb.save(fname)
def dataDownload(): #function to parse thru the online table and read the data
data = driver.find_elements_by_class_name("rowNumber")
for td in data:
values = td.text
print(td)
excelFactory(values, row, col)
NxtButton = driver.find_elements_by_class_name('NxtButtonClassName')
for content in NxtButton:
if content.text == 'Next':
content.click()
dataDownload()
dataDownload()
Related
I am writing an Excel macro, using Selenium basic, to download data from a web-based tool.
Using Selenium, I need to select 200 cells in a column in Excel and send them to a text box in a webpage.
The following line does this with the single cell A1:
driver.FindElementById("batch_requests").SendKeys [A1]
How can I send the entire range A1:A200?
Looping through and doing one at a time would be too time-consuming, as there are thousands of data points I need to paste 200 at a time.
Using standard sendkeys in VBA (not Selenium) is not a good solution, because the computer will be in use while the macros run in the background.
EDIT - Qharr's answer fitted to my needs
Not the most elegant solution, but it works every time in the situation(s) I will be using it.
' Copy column A 200 cells at a time into webpage text input field
Dim clipboard As Object
Dim data As String
Dim myRange As Range
Set clipboard = GetObject("New:{1C3B4210-F441-11CE-B9EA-00AA006B1A69}")
' See how many rows of data in column A
numRows = Range("A1", Range("A1").End(xlDown)).Rows.Count
' Step through column A, 200 cells at a time (until numRows is exceeded)
' and create a range to be copied to clipboard and sent to webpage via Selenium
' 'Batch_Requests' is ID of input field page element to send text to.
For x = 2 To numRows Step 200
Set myRange = Range("A" & x & ":A" & (x + 200))
myRange.Copy
With clipboard
.GetFromClipboard
data = .GetText
driver.FindElementById("batch_requests").SendKeys data
End With
' *****************************************************************************
' Insert Seleneium code here to manipulate the web page with the data
' *****************************************************************************
' Clear text input field
driver.FindElementById("batch_requests").Clear
' Go to next 200 in column A (until none left)
Next x
This is sendKeys based. Not so bad with selenium in my opinion. I am using the clipboard to generate the text from range to paste
Option Explicit
Public Sub PasteInfo()
Dim d As WebDriver, clipboard As Object, data As String
Set d = New ChromeDriver
Const url = "https://codebeautify.org/Xpath-Tester"
With d
'.AddArgument "--headless"
.Start "Chrome"
.get url
[A1:A3].Copy
Set clipboard = GetObject("New:{1C3B4210-F441-11CE-B9EA-00AA006B1A69}")
With clipboard
.GetFromClipboard
data = .GetText
End With
.FindElementById("xmlString").SendKeys data
Stop '<=Delete me later
.Quit
End With
End Sub
If you want to send value by selenium then
first go to the web page and inspect the web page and find the name or id of the input field
you can send values by the id or name of the input field
then go to your vba editor
if you aware of vba selenium basic then
Object_Of_chromedriver.findelmentbyname("NameOfInputField").sendkeys("the value")
I have many excel files of the same structure in one folder (Sample 1, Sample 2......Sample 20). I created another excel file in the same folder that needs to pull out information from each other excel file (Results). There is a specific column in each Sample file that I need to copy and paste into a row in the Results file. I am trying to create a tool or Macro that can, from a push of a button, extract the same column from each file and paste it into a new row in the Results file. I cannot alter anything in the Sample files and this should be done automatically without opening each file. Also new Sample files will be added to the folder (Sample 21...22 etc) so the function should be able to pull from the new files.
Edit.
Based off of Pomul's suggestion of transposing the rows. I came up with the following code and results. Right now I am testing the code to transpose in the same worksheet:
Transpose Image Screenshot
Please let me know why my code makes another column instead of transposing it into a row.
This seems to work:
Sub Button1_click()
Dim i&, z&, x&
i = Worksheets("Sheet2").Cells(Rows.Count, "B").End(xlUp).Row
z = 1: x = 1
While z <= i
Worksheets("Sheet1").Range("A" & x).Resize(, i) = _
WorksheetFunction.Transpose(Worksheets("Sheet2").Range("B" & z).Resize(i))
z = i + 1
Wend
End Sub
so lets say I have this query for example
SELECT *
FROM dbo.test
WHERE (person_ID IN ('person1', 'person2', 'person3', 'person4',
'person5', 'person6', 'person7', 'person8', 'person9',
'person10', 'person11', 'person12', 'person13',
'person14', 'person15', 'person16', 'person17',
'person18', 'person19', 'person20'))
It gives me all the results for these values in the test table right. I need to save the results for each person into its own excel file or into it's own tab within excel. Instead of writing a query with them all together.
Basically how would I write this query properly so it would give me separate outputs for each person that I could quickly copy and paste into its own excel sheet?
The difficult part for me is I have lets say close to 200+ values in an excel sheet right now that I will be searching against (for example with the above query person1 to 200) It's all in a column so I just formatted all the values with '#', so it would come out as example 'person1', that I just copy and pasted into a whereIN( Clause. If I want to do it separately whats the best way to do it so it will give me a lot of different results back all split up I could copy and paste best with the select all as each person_id could come back with a lot of results.
Also is copy and pasting the easiest way in the end ? For them to go into their own excel file or is there an easier way?
For something like this I would suggest SSRS. Here is an example that shows how to use Reporting Services to export an Excel file with Multiple Worksheets in SQL Server 2008 R2 using SSRS Report Designer in Visual Studio 2010.
Here's a sample that you can try. It's not the easiest way for everyone, but it is a way that I felt was relatively easy.
I have used MS Access 2013 in this sample of convenience instead of SQL Server 2014 or other real, modern RDBMS. You should be able to take this example and adapt for your needs
I created an MS Access file c:\test\Database1.accdb. It contains one table called Table1 with following fields:
ID
Firstname
Lastname
Age
Data looks like this:
ID Firstname Lastname Age
--- ---------- --------- ----
1 John Smith 12
2 Matt Johnson 21
Then, I created an Excel sheet and saved it as Macro enabled Excel file (Database1.xlsm).
Here're the sequence of events thereafter:
Name first sheet 1
If there are any other sheets, delete them
Go to Data > From Other Sources > From Microsoft Query > MS Access Database > Navigate to c:\test and choose Database1.accdb
Choose Table 1 and use the > arrow sign to add all fields > Next. Next again. Next again
Select View data or edit query in Microsoft Query. Click Finish
Click SQL button
Ensure that the SQL looks like this (edit the SQL if necessary)
SELECT Table1.ID, Table1.Firstname, Table1.Lastname, Table1.Age
FROM C:\test\Database1.accdb.Table1 Table1
where ID = ?
Click OK. You will be prompted with a dialog box titled Enter Parameter Value. Type 1 and click OK
X out of (i.e. close out) Microsoft Query
Import data dialog box will appear with Existing worksheet radio button already selected and =$A$1 entered. Change that to =$A$3
Right click column A3 > Table > Parameters
In the Parameters dialog box, select radio button Get the value from the following cell and type in that text box =A1
Go to File > Options > Customize Ribbon > check Developer check box on the right side and click OK (See instructions if needed)
Go to Developer > Visual Basic
On top right, click on VBAProject (Database1.xlsm)
From the application menu on the top, click Insert > Module. Module folder will be created with Module1 in it
Write the following Sub in it
Sub Macro1()
Dim StartNumber As Integer
Dim EndNumber As Integer
StartNumber = 2
EndNumber = 10
' remove all sheets except the first sheet,
' assuming first sheet is called 1
Dim ws As Worksheet
Application.DisplayAlerts = False
For Each ws In ActiveWorkbook.Sheets
If Not ws.Name = "1" Then
Sheets("1").Select
ws.Delete
End If
Next
Application.DisplayAlerts = True
' make a copy of sheet1 that we prepared carefully
Dim i As Integer
For i = StartNumber To EndNumber
Sheets("1").Select
Sheets("1").Copy After:=Sheets(i - 1)
Sheets(i).Select
Sheets(i).Name = i
Next
' update cell A1 of each sheet
For Each ws In ActiveWorkbook.Sheets
ws.Cells(1, 1) = ws.Name
Next
' refresh query in all sheets
ActiveWorkbook.RefreshAll
End Sub
Save the file. Close Microsoft Visual Basic for Applications window
Go to Developer toolbar > Macro > Click Macro1 > Options > Type Shift + t (capital T) in the text box to invoke Macro1 with CTRL+SHIFT+t combination
Press that combination
You should now see 10 sheets named 1 through 10. Each sheet's cell A1 will contain the name of the sheet. Sheets will refresh automatically as well.
If you run it again, sheet named 1 will be untouched. All other sheets will be wiped off and recreated.
Give it a try. Once successful, delete all sheets except 1. Switch your connection to SQL Server and test Sheet 1. After it works well, just press CTRL + SHIFT + T and you will have multiple sheets with the data you are looking for.
I have been trying to figure out how to do this for a little while.. Here is my situation:
I have an Excel spreadsheet (lets call this "workbook.xls" and I have two sheets that I need to use for my script. One sheet has a column of data "columnA" and I am trying to put cell A2 (A1 is a header) into my QTP script to use.
I want to be able to use this specific value and then process some data and then use the next cell, A3 to process another procedure/process, and so on.. Is this possible?
In other languages, I know that I could use a for loop to iterate through the data, but I am getting stuck on trying to process the data and then going back through the loop to get the data.
For example:
Excel may have A2 as "1" and A3 as "2". I want to be able to put the value of A1 into an input field on the browser.. Then I want to be able to do some actions (clicking, writing to another file, etc) with this value.. Then I want to move onto the value A3 and put the value of A3 into that same input field as before and then do some other actions (that may not be what I did with A2). I am wondering how I could achieve this scenario.
Thanks in advance.
Yes, it's possible. Assuming I understood your question correctly you could do something like this:
Set xl = CreateObject("Excel.Application")
xl.Visible = True 'set to False for production
Set wb = xl.Workbooks.Open("C:\path\to\workbook.xls")
Set ws = wb.Sheets(1)
firstRow = ws.UsedRange.Rows(1).Row
lastRow = firstRow - 1 + ws.UsedRange.Rows.Count
For i = firstRow+1 To lastRow
val = ws.Cells(i, 1).Value 'get value from cell
'do stuff with val
Next
wb.Close
xl.Quit
A test's DataTable is an Excel spreadsheet. You could import your data into your test's DataTable in QTP - in the default sheet(s) or add your own - and then access them in code:
Set sheet = DataTable.GetSheet "Sheet Name"
numRows = sheet.GetRowCount
For index = 0 To numRows
value = sheet.GetParameter(index)
'Do something
Next
I'm looking for the equivalent VBA-GAS of:
Application.ScreenUpdating = False
I'm running a very long macro in one of my Google Spreadsheets and it takes at least 30 seconds to finish it every time. It would be helpful if the macro didn't refresh the screen after every line of code.
There isnt. However you should batch all setValues into range writes (ideally a single range write) which will help here.Only call SpreadsheetApp.flush() at the very end.
I ended up creating a second sheet called 'Start' which just had A1='Please wait...'. The sheet I display all my data on is called 'Report'
Then used:
//Hide Report sheet
SpreadsheetApp.getActive().getSheetByName("Start").showSheet();
var sheet = SpreadsheetApp.getActive().getSheetByName("Report");
sheet.hideSheet();
//Code to update sheet here
sheet.appendRow('Blah','Blah','Blah');
//Show Report sheet
sheet.showSheet();
SpreadsheetApp.getActive().setActiveSheet(sheet);
SpreadsheetApp.getActive().getSheetByName("Start").hideSheet();
The way I got around this is by bypassing every instance of "setActiveSheet" or "activate."
For example:
var spreadsheet = SpreadsheetApp.getActive();
spreadsheet.getRange('A2').activate();
spreadsheet.setActiveSheet(spreadsheet.getSheetByName('Sheet2'), true);
spreadsheet.getRange('B4').activate();
spreadsheet.getRange('Sheet1!A2').copyTo(spreadsheet.getActiveRange(), SpreadsheetApp.CopyPasteType.PASTE_NORMAL, false);
can be shortened to:
var spreadsheet = SpreadsheetApp.getActive();
spreadsheet.getRange('Sheet1!A2').copyTo(spreadsheet.getRange('Sheet2!B4'), SpreadsheetApp.CopyPasteType.PASTE_NORMAL, false);
and can be run from any sheet on the document without changing the selected sheet or cell.
There may be a reason for Google to automatically add this redundancy, but this does appear to work in every case I've used it in.
While I understand that the below does not necessarily speed up the macro, it makes for a better UX and is important to note.
For many function calls and updates, VBA first requires that we "unhide" the respective sheet:
' Unhide the sheet, write to it, hide it '
Dim lastrow As Long
Sheets("Report").Visible = True
Sheets("Report").Select
' Append a Row '
lastrow = ActiveSheet.Cells(Rows.Count, "A").End(xlUp).Row + 1
ActiveSheet.Cells(lastrow, "A").Value = "Blah"
Sheets("Report").Select
ActiveWindow.SelectedSheets.Visible = False
The same is not required for many of the Google Apps Script Classes:
// Hide the sheet, write to it, show it
var sheet = SpreadsheetApp.getActive().getSheetByName('Report').hideSheet();
sheet.appendRow(['Blah']);
sheet.showSheet();
If your script involves looping through some range and reading and/or writing values, it will always be slow as every read/write is a separate API call.
Instead of looping through 10 rows and reading a cell value like in excel VBA, you should specify the range with getRange() and get all values by one call, which can be done with getValues(). It returns a 2d array with values from specified range.
You can do all operations you need on that data and you can return values with setValues()