I have problems with my records within my database, so I have a template with about 260,000 records and for each record they have 3 identification columns to determine what time period the record is from and location: one for year, one for month, and one for region. Then the information for identifying the specific item is TagName, and Description. The Problem I am having is when someone entered data into this database they entered different description for the same device, I know this because the tag name is the same. Can I write code that will go through the data base find the items with the same tag name and use one of the descriptions to replace the ones that are different to have a more uniform database. Also some devices do not have tag names so we would want to avoid the "" Case.
Also moving forward into the future I have added more columns to the database to allow for more information to be retrieved, is there a way that I can back fill the data to older records once I know that they have the same tag name and Description once the database is cleaned up? Thanks in advance for the information it is much appreciated.
I assume that this will have to be done with VBA of some sort to modify records by looking for the first record with that description and using a variable to assign that description to all the other items with the same tag name? I just am not sure of the correct VBA syntax to go about this. I assume a similar method would be used for the backfilling process?
Your question is rather broad and multifaceted, so I'll answer key parts in steps:
The Problem I am having is when someone entered data into this
database they entered different description for the same device, I
know this because the tag name is the same.
While you could fix up those inconsistencies easily enough with a bit of SQL code, it would be better to avoid those inconsistencies being possible in the first place:
Create a new table, let's call it 'Tags', with TagName and TagDescription fields, and with TagName set as the primary key. Ensure both fields have their Required setting to True and Allow Zero Length to False.
Populate this new table with all possible tags - you can do this with a one-off 'append query' in Access jargon (INSERT INTO statement in SQL).
Delete the tag description column from the main table.
Go into the Relationships view and add a one-to-many relation between the two tables, linking the TagName field in the main table to the TagName field in the Tags table.
As required, create a query that aggregates data from the two tables.
Also some devices do not have tag names so we would want to avoid the
"" Case.
In Access, the concept of an empty string ("") is different from the concept of a true blank or 'null'. As such, it would be a good idea to replace all empty strings (if there are any) with nulls -
UPDATE MyTable SET TagName = Null WHERE TagName = '';
You can then set the TagName field's Allow Zero Length property to False in the table designer.
Also moving forward into the future I have added more columns to the
database to allow for more information to be retrieved
Think less in terms of more columns than more tables.
I assume that this will have to be done with VBA of some sort to modify records
Either VBA, SQL, or the Access query designers (which create SQL code behind the scenes). In terms of being able to crunch through data the quickest, SQL is best, though pure VBA (and in particular, using the DAO object library) can be easier to understand and follow.
I'm trying to create a sub-table from another table of all the last name fields sorted A-Z which have a phone number field that isn't null. I could do this pretty easy with SQL, but I have no clue how to go about running a SQL query within Excel. I'm tempted to import the data into postgresql and just query it there, but that seems a little excessive.
For what I'm trying to do, the SQL query SELECT lastname, firstname, phonenumber WHERE phonenumber IS NOT NULL ORDER BY lastname would do the trick. It seems too simple for it to be something that Excel can't do natively. How can I run a SQL query like this from within Excel?
There are many fine ways to get this done, which others have already suggestioned. Following along the "get Excel data via SQL track", here are some pointers.
Excel has the "Data Connection Wizard" which allows you to import or link from another data source or even within the very same Excel file.
As part of Microsoft Office (and OS's) are two providers of interest: the old "Microsoft.Jet.OLEDB", and the latest "Microsoft.ACE.OLEDB". Look for them when setting up a connection (such as with the Data Connection Wizard).
Once connected to an Excel workbook, a worksheet or range is the equivalent of a table or view. The table name of a worksheet is the name of the worksheet with a dollar sign ("$") appended to it, and surrounded with square brackets ("[" and "]"); of a range, it is simply the name of the range. To specify an unnamed range of cells as your recordsource, append standard Excel row/column notation to the end of the sheet name in the square brackets.
The native SQL will (more or less be) the SQL of Microsoft Access. (In the past, it was called JET SQL; however Access SQL has evolved, and I believe JET is deprecated old tech.)
Example, reading a worksheet: SELECT * FROM [Sheet1$]
Example, reading a range: SELECT * FROM MyRange
Example, reading an unnamed range of cells: SELECT * FROM [Sheet1$A1:B10]
There are many many many books and web sites available to help you work through the particulars.
Further notes
By default, it is assumed that the first row of your Excel data source contains column headings that can be used as field names. If this is not the case, you must turn this setting off, or your first row of data "disappears" to be used as field names. This is done by adding the optional HDR= setting to the Extended Properties of the connection string. The default, which does not need to be specified, is HDR=Yes. If you do not have column headings, you need to specify HDR=No; the provider names your fields F1, F2, etc.
A caution about specifying worksheets: The provider assumes that your table of data begins with the upper-most, left-most, non-blank cell on the specified worksheet. In other words, your table of data can begin in Row 3, Column C without a problem. However, you cannot, for example, type a worksheet title above and to the left of the data in cell A1.
A caution about specifying ranges: When you specify a worksheet as your recordsource, the provider adds new records below existing records in the worksheet as space allows. When you specify a range (named or unnamed), Jet also adds new records below the existing records in the range as space allows. However, if you requery on the original range, the resulting recordset does not include the newly added records outside the range.
Data types (worth trying) for CREATE TABLE: Short, Long, Single, Double, Currency, DateTime, Bit, Byte, GUID, BigBinary, LongBinary, VarBinary, LongText, VarChar, Decimal.
Connecting to "old tech" Excel (files with the xls extention): Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\MyFolder\MyWorkbook.xls;Extended Properties=Excel 8.0;. Use the Excel 5.0 source database type for Microsoft Excel 5.0 and 7.0 (95) workbooks and use the Excel 8.0 source database type for Microsoft Excel 8.0 (97), 9.0 (2000) and 10.0 (2002) workbooks.
Connecting to "latest" Excel (files with the xlsx file extension): Provider=Microsoft.ACE.OLEDB.12.0;Data Source=Excel2007file.xlsx;Extended Properties="Excel 12.0 Xml;HDR=YES;"
Treating data as text: IMEX setting treats all data as text. Provider=Microsoft.ACE.OLEDB.12.0;Data Source=Excel2007file.xlsx;Extended Properties="Excel 12.0 Xml;HDR=YES;IMEX=1";
(More details at http://www.connectionstrings.com/excel)
More information at http://msdn.microsoft.com/en-US/library/ms141683(v=sql.90).aspx, and at http://support.microsoft.com/kb/316934
Connecting to Excel via ADODB via VBA detailed at http://support.microsoft.com/kb/257819
Microsoft JET 4 details at http://support.microsoft.com/kb/275561
tl;dr; Excel does all of this natively - use filters and or tables
(http://office.microsoft.com/en-gb/excel-help/filter-data-in-an-excel-table-HA102840028.aspx)
You can open excel programatically through an oledb connection and execute SQL on the tables within the worksheet.
But you can do everything you are asking to do with no formulas just filters.
click anywhere within the data you are looking at
go to data on the ribbon bar
select "Filter" its about the middle and looks like a funnel
you will have arrows on the tight hand side of each cell in the the first row of your table now
click the arrow on phone number and de-select blanks (last option)
click the arrow on last name and select a-z ordering (top option)
have a play around.. some things to note:
you can select the filtered rows and pasty them somewhere else
in the status bar on the left you will see how many rows meet you filter criteria out of the total number of rows. (e.g. 308 of 313 records found)
you can filter by color in excel 2010 on wards
Sometimes i create calculated columns that give statuses or cleaned versions of data you can then filter or sort by theses too. (e.g. like the formulae in the other answers)
DO it with filters unless you are going to do it a lot or you want to automate importing data somewhere or something.. but for completeness:
A c# option:
OleDbConnection ExcelFile = new OleDbConnection( String.Format( "Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties=\"Excel 12.0;HDR=YES\"", filename));
ExcelFile.Open();
a handy place to start is to take a look at the schema as there may be more there than you think:
List<String> excelSheets = new List<string>();
// Add the sheet name to the string array.
foreach (DataRow row in dt.Rows) {
string temp = row["TABLE_NAME"].ToString();
if (temp[temp.Length - 1] == '$') {
excelSheets.Add(row["TABLE_NAME"].ToString());
}
}
then when you want to query a sheet:
OleDbDataAdapter da = new OleDbDataAdapter("select * from [" + sheet + "]", ExcelFile);
dt = new DataTable();
da.Fill(dt);
NOTE - Use Tables in excel!:
Excel has "tables" functionality that make data behave more like a table.. this gives you some great benefits but is not going to let you do every type of query.
http://office.microsoft.com/en-gb/excel-help/overview-of-excel-tables-HA010048546.aspx
For tabular data in excel this is my default.. first thing i do is click into the data then select "format as table" from the home section on the ribbon. this gives you filtering, and sorting by default and allows you to access the table and fields by name (e.g. table[fieldname] ) this also allows aggregate functions on columns e.g. max and average
Might I suggest giving QueryStorm a try - it's a plugin for Excel that makes it quite convenient to use SQL in Excel.
In the SQL scripts Excel tables are visible as if they were regular database tables.
All four SQL data operations are supported: select/update/insert/delete.
The engine that executes the queries is SQLite so you can use joins, common table expressions, window functions, etc... And you get the fancy stuff like code completion, auto-formatting, symbol tooltips etc...
It has a completely free community edition for use by individuals and small companies. If you're in a company that has more than 5 employees or more than $1M in yearly revenue, you'll need a paid license but you can use a free trial key for evaluation purposes.
This blog post describes the SQL functionality of the plugin in much more detail.
Disclaimer: I'm the author.
You can do this natively as follows:
Select the table and use Excel to sort it on Last Name
Create a 2-row by 1-column advanced filter criteria, say in
E1 and E2, where E1 is empty and E2 contains the formula =C6=""
where C6 is the first data cell of the phone number column.
Select the table and use advanced filter, copy to a range, using
the criteria range in E1:E2 and specify where you want to copy the
output to
If you want to do this programmatically I suggest you use the Macro Recorder to record the above steps and look at the code.
The accepted answers here are old technology and shouldn't be attempted.
Back when this question was written, Power Query wasn't a well known option and wasn't available unless you were on the latest version of Office and installed it as a separate Add-in.
Now, Power Query is included in Excel and used by default to get data. It is the right way to do this. It is simple, fast and effective.
Here is the answer to the question in Power Query. Search on "getting started with Power Query" if you need help replicating this. Once you get started with Power Query, you'll see this is very basic and easy to do with the Advanced Editor:
let
Source = Excel.CurrentWorkbook(){[Name="Names"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"lastname", type text}, {"firstname", type text}, {"phonenumber", type text}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([phonenumber] <> null)),
#"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"lastname", "firstname", "phonenumber"}),
#"Sorted Rows" = Table.Sort(#"Removed Other Columns",{{"lastname", Order.Ascending}})
in
#"Sorted Rows"
You can use SQL in Excel. It is only well hidden.
See this tutorial:
http://smallbusiness.chron.com/use-sql-statements-ms-excel-41193.html
If you need to do this once just follow Charles' descriptions, but it is also possible to do this with Excel formulas and helper columns in case you want to make the filter dynamic.
Lets assume you data is on the sheet DataSheet and starts in row 2 of the following columns:
A: lastname
B: firstname
C: phonenumber
You need two helper columns on this sheet.
D2: =if(A2 = "", 1, 0), this is the filter column, corresponding to your where condition
E2: =if(D2 <> 1, "", sumifs(D$2:D$1048576, A$2:A$1048576, "<"&A2) + sumifs(D$2:D2, A$2:A2, A2)), this corresponds to the order by
Copy down these formulas as far as your data goes.
On the sheet which should display your result create the following columns.
A: A sequence of numbers starting with 1 in row 2, this limits the total number of rows you can get (kind like a limit in sequel)
B2: =match(A2, DataSheet!$E$2:$E$1048576, 0), this is the row of the corresponding data
C2: =iferror(index(DataSheet!A$2:A$1048576, $B2), ""), this is the actual data or empty if no data exists
Copy down the formulas in B2 and C2 and copy-past column C to D and E.
If you have GDAL/OGR compiled with the against the Expat library, you can use the XLSX driver to read .xlsx files, and run SQL expressions from a command prompt. For example, from a osgeo4w shell in the same directory as the spreadsheet, use the ogrinfo utility:
ogrinfo -dialect sqlite -sql "SELECT name, count(*) FROM sheet1 GROUP BY name" Book1.xlsx
will run a SQLite query on sheet1, and output the query result in an unusual form:
INFO: Open of `Book1.xlsx'
using driver `XLSX' successful.
Layer name: SELECT
Geometry: None
Feature Count: 36
Layer SRS WKT:
(unknown)
name: String (0.0)
count(*): Integer (0.0)
OGRFeature(SELECT):0
name (String) = Red
count(*) (Integer) = 849
OGRFeature(SELECT):1
name (String) = Green
count(*) (Integer) = 265
...
Or run the same query using ogr2ogr to make a simple CSV file:
$ ogr2ogr -f CSV out.csv -dialect sqlite \
-sql "SELECT name, count(*) FROM sheet1 GROUP BY name" Book1.xlsx
$ cat out.csv
name,count(*)
Red,849
Green,265
...
To do similar with older .xls files, you would need the XLS driver, built against the FreeXL library, which is not really common (e.g. not from OSGeo4w).
You can experiment with the native DB driver for Excel in language/platform of your choice. In Java world, you can try with http://code.google.com/p/sqlsheet/ which provides a JDBC driver for working with Excel sheets directly. Similarly, you can get drivers for the DB technology for other platforms.
However, I can guarantee that you will soon hit a wall with the number of features these wrapper libraries provide. Better way will be to use Apache HSSF/POI or similar level of library but it will need more coding effort.
Microsoft Access and LibreOffice Base can open a spreadsheet as a source and run sql queries on it. That would be the easiest way to run all kinds of queries, and avoid the mess of running macros or writing code.
Excel also has autofilters and data sorting that will accomplish a lot of simple queries like your example. If you need help with those features, Google would be a better source for tutorials than me.
I might be misunderstanding me, but isn't this exactly what a pivot table does? Do you have the data in a table or just a filtered list? If its not a table make it one (ctrl+l) if it is, then simply activate any cell in the table and insert a pivot table on another sheet. Then Add the columns lastname, firstname, phonenumber to the rows section. Then Add Phone number to the filter section and filter out the null values. Now Sort like normal.
I have come across the problem of generating complex access reports (by complex I mean with data processing, variable number of fields, among others).
Let me explain in deeper detail some of the things I need to implement:
Some fields should not show according to some values in a query
If a certain record does not exist, a nice colored (very noticeable) message should appear instead of the values that would be there (Suppose, for example, that a record with 03/04/2009 in the date field exists, a record with 03/06/2009 in the date field also exists but no record with 03/05/2009 exists. Before showing the data related to the last record, I should print something like "Didn't show up on 03/05/2009")
A bar chart that takes as data not the values in the records, but instead something else that is calculated over a set of records (like an average of all grades for a certain date). The number of series in this chart also varies according to values in the records, this chart would not be in the detail section, but instead in the page heading or some kind of group heading.
It should also be mentioned that the query is a TRANSFORM query (more precisely, an INNER JOIN of many TRANSFORM queries), and thus the number of columns returned by the query varies. While in the past I've been unable to bind this query as the recordsource for the report, somehow Access stopped complaining for now (can someone please clarify this? Is this normal, should I not worry about it and use it as a recordsource or should I avoid it?)
There are two options to achieve what I want (that I can see for now):
Create a report with no record source and lots of unbound fields, and through several events (Report_Open, Section_Format, etc.) and with the help of DAO, manually set the values of these fields. Changing the Data Series of the chart is also possible through VBA.
Set the record source to the query, and create some crazy and confusing VBA code to deal with the data and implement everything I need.
It seems to me that option 2 is going to be a huge headache and waste of time, and I recognize option 1 is pretty much like writing to an Excel file (since all the data is obtained with DAO), which would be much easier since I have much more control over almost everything there (but for many other reasons, we want everything in an access report)
While I'm biased and intend to go with option 1, I have found several problems with this option, for example:
I can't find a way to create new pages in the report with VBA, and thus I'm limited only to the first page.
Lack of some kind of free, online, decent and complete documentation on VBA and Access Reports
Also, if option 2 is more viable, I'm certainly willing to go with it, but I would also need some advice, and perhaps some tips to solving the problems I mentioned in this question.
So, the questions are:
Where can I find some decent and complete documentation on Access Reports and VBA?
How can I create pages in an access report, and choose which page I want to write to?
With the problem I have in my hands, will I reach any bottlenecks I should know about? Should I already be thinking of alternatives to Access Reports (writing to a spreadsheet, for example?)
Sounds like you want to dynamically create the report and avoid all the dummy text boxes.
In regard to:
I can't find a way to create new pages
in the report with VBA, and thus I'm
limited only to the first page.
Your solution #1 seems to assume an unbound report.
I think what I'd do is have the form the crosstab as the rowsource, so you'd have records to generate the pages, and then define your report's controls with no ControlSource (except for the controls that are bound to fields that are always present in the CrossTab). Then you could assign the ControlSources at runtime based on the particular columns. Here's the SQL for a crosstab grabbed from an app I'm working on now:
TRANSFORM First(impNoMatch.PersonID) AS FirstOfPersonID
SELECT impNoMatch.LastName, impNoMatch.FirstBame
FROM impNoMatch
GROUP BY impNoMatch.LastName, impNoMatch.FirstName
PIVOT impNoMatch.Status;
Now, you know that the fields in the SELECT clause will always be present, so if you opened a recordset on the SQL string you are using and count the number of fields in the recordset's Fields collection (you can't use the report's Recordset unless it's an ADO recordset, i.e., not bound to the Recordsource):
Dim strSQL As String
Dim rsFields As DAO.Recordset
Dim lngFieldCount As Long
strSQL = Me.Recordsource
Set rsFields = CurrentDB.OpenRecordset(strSQL)
lngFieldCount = rsFields.Fields.Count
From that, since you know the number of fields in the SELECT statement (i.e., the row headings), you can calculate the number of dynamic controls you want to assign, and you can use this recordset's fields collection to assign the ControlSources and unhide the controls.
You'd start out with all your controls that will display the dynamic fields set so their Visible property is FALSE. You'd also use a naming convention for those controls. In the code below, I've used txtNN, where NN is the numeric index in the Fields collection formatted as 2 digits. Here's the code (which adds lines to what's listed above, and is run in the OnOpen event):
Dim strSQL As String
Dim rsFields As DAO.Recordset
Dim lngFieldCount As Long
Dim l As Long
Dim strControlName As String
strSQL = Me.RecordSource
Set rsFields = CurrentDb.OpenRecordset(strSQL)
lngFieldCount = rsFields.Fields.Count
For l = 2 To lngFieldCount - 1
strControlName = "txt" & Format(l, "00")
Me(strControlName).ControlSource = rsFields.Fields(l).Name
Me(strControlName).Visible = True
Next l
rsFields.Close
Set rsFields = Nothing
Now, if you want to get fancy, you can reformat the controls, changing widths and horizontal/vertical position. If you do that, you have to do it in a different event, and it's a bit tricky to choose that. The only good place to put it is in a report group's header's OnFormat event. If you don't have any grouping, you can add one that doesn't do anything. In the case of my crosstab, a two-level sort on Lastname and Firstname and a header on the Firstname group with nothing in it is a good place to use the OnFormat event to change the appearance/layout of the controls on your report.
As to your question about how to learn how to do this, I recommend picking up an intermediate/advance Access programming book. The Access Developers Handbook is the gold standard on this, and includes tons of examples of programmatic control of reports.
hey guys, could someone show me the simple update query through vb? I need to add new fields to the table (just 3) and add a couple text boxes on a form so that users can add some additional data relative to the record (which is already what this form is based on).
So the first form I have is a form that populates a list, when the user double clicks on a selection from that list, it opens a new form, so that the ID of the the table that is tied to this form that I need to add the these text boxes on (all the combo boxes and text boxes relative to one record are tied to the active form at this point, however there are all unbound. On a button click there is already vb that saves the information to the table). I did not create this however, it was built by someone who is not there anymore, and apparently is better than I at this stuff. My problem is that there is soooo much vb that checks for records, and various sql statements based on case, that I cannot decipher it to its simplest form.
So I was looking for a simple example of an update sql statement in vb so I can try to break this apart.
I need it to update the record based on the ID: sql WHERE RecordID = me.RecordID
I actually thought I knew how to do this based on examples, however every time I try, then try to run on button click, I get a run-time error of SYNTAX error, and the debug just highlights the db.execute(sql) part. So I tried to get the resulting immediate window of the sql statement, and it looks fine to me:
UPDATE tblMain
SET [Name] = "John Doe",
[DATE] = #9/30/2009#,
[TYPE] = "TypeA",
WHERE RecordID = 958;
Can I update a table without accounting for every field in the table (because this one has about 15 plus the new 3, so I am ignoring about 14 fields here, but I do not want to alter those anyway???
So as always, I appreciate the help yall!! Thanks!
EDIT:
Sorry I always forget this....I was actaully trying it DAO....
Dim db as DAO.Database
Dim sql as String
set db = CurrentDb
etc
You were thaaat close! You have a simple extra comma after your last column. Get rid of it and it works fine.
UPDATE tblMain SET
[Name] = "John Doe",
[DATE] = #9/30/2009#,
[TYPE] = "TypeA"
WHERE RecordID = 958;
Yes, you can absolutely update only a few columns rather than all of them. That is a best practice, BTW.
Finally, It's considered bad practice to name your columns after reserved words like "Name" and "Date", but I know you inherited this.
You were wise to include Debug.Print sql in your code. bpayne already pointed out the extra comma in your SQL statement.
I want to point out another trouble shooting technique you may find useful to debug SQL statement problems.
Copy the statement from the Immediate Window, and paste it into the SQL View of a new query. Modify the query in the query designer until you can get it working, then revise your VBA code to generate a matching SQL statement.
In this case you might not have noticed the extra comma. However, you could create another new query and build the UPDATE statement from scratch in the query designer. After getting that one working, you could compare its SQL View to the failing query.