I am using asp.net core and Razor - and as it is a help system I would like to implement some kind of search facility to bring back a list of results hyperlinked based on the search terms.
I would like the search to iterate essentially over the content contained within the and tags and then link this to the appropriate page/view.
What is the best way to do this?
I'm not even sure how you get a handle on the actual content of your own cshtml pages and then go from there.
This question is far too broad. However, I can provide you some pointers.
First, you need to determine what you're actually wanting to surface and where that data lives. Your question says "static web pages", but then you mention .cshtml. Traditionally, when it comes to creating your own search, you're going to have access to some particular dataset (tables in a database, for example). It's much simpler to search across the more structured data than the end result of it being dumped in various and sundry places over a web page.
Search engines like Google only index in this way because they typically don't have access to the raw data (although some amount of "access" can be granted via things like JSON-LD and other forms of Schema.org markup). In other words, they actually read from the web page out of necessity, because that's what they have to work with. It's certainly not the approach you would take if you have access to the data directly.
If for some reason you need to actually spider an index your own site's HTML content, then you'll essentially have to do what the big boys do: create a bot, run it on a schedule, crawl your site, link by link, downloading each document, and then parse and process it. The end result would be to create a set of structured data that you can actually query against, which is why all this is pretty much just wasted effort if you already have that data.
Once you have the data, however you got there, you simply query it. In the most basic of forms, you could store it in a table in a database and literally issue SQL queries against it. Your search keywords/parameters are essentially the WHERE of the SELECT statement, so you'd have to figure out a way to map the keywords/parameters you're receiving to an acceptable WHERE clause that achieves that.
More traditionally, you'd use an actual search engine: essentially a document database that is designed and optimized for search, and generally provides a more search-appropriate API to query against. There's lots of options in this space from roll your own to hosted SaaS solutions, and anywhere in between. Of course the cost meter goes down the more work you have to do and goes up the more out of the box it is.
One popular open-source and largely free option is Elasticsearch. It uses Lucene indexes, which it stitches to together in a clustered environment to provide failover and scale. Deployment is a beast, to say the least, though it's gotten considerably better with things like containerization and orchestration. You can stand up an Elasticsearch cluster in something like Kubernetes with relative ease, though you still will probably need to do a bit of config. Elasticsearch does also have hosted options, but you know, cost.
I have an application that allows the user to drill down through data from a single large table with many columns. It works like this:
There is a list of distinct top-level table values on the screen.
User clicks on it, then the list changes to the distinct next-level values for whatever was clicked on.
User clicks on one of those values, taken to 3rd level values, etc.
There are about 50 attributes they could go through, but it usually ends up only being 3 or 4. But since those 3 or 4 vary among the 50 possible attributes, I have to persist the selections to the browser. Right now I do it in a hideous and bulky hidden form. It works, but it is delicate and suboptimal. In order for it to work, the value of whatever level attribute is on the screen is populated in the appropriate place on the hidden form on the click event, and then a jQuery Ajax POST submits the form. Ugly.
I have also looked at Backbone.js, but I don't want to roll another toolkit into this project while there may be some other simple convention that I'm missing. Is there a standard Rails Way of doing something like this, or just some better way period?
Possible Approaches to Single-Table Drill-Down
If you want to perform column selections from a single table with a large set of columns, there are a few basic approaches you might consider.
Use a client-side JavaScript library to display/hide columns on demand. For example, you might use DataTables to dynamically adjust which columns are displayed based on what's relevant to the last value (or set of values) selected.
You can use a form in your views to pass relevant columns names into the session or the params hash, and inspect those values for what columns to render in the view when drilling down to the next level.
Your next server-side request could include a list of columns of interest, and your controller could use those column names to build a custom query using SELECT or #pluck. Such queries often involve tainted objects, so sanitize that input thoroughly and handle with care!
If your database supports views, users could select pre-defined or dynamic views from the next controller action, which may or may not be more performant. It's at least an idea worth pursuing, but you'd have to benchmark this carefully, and make sure you don't end up with SQL injections or an unmanageable number of pre-defined views to maintain.
Some Caveats
There are generally trade-offs between memory and latency when deciding whether to handle this sort of feature client-side or server-side. It's also generally worth revisiting the business logic behind having a huge denormalized table, and investigating whether the problem domain can't be broken down into a more manageable set of RESTful resources.
Another thing to consider is that Rails won't stop you from doing things that violate the basic resource-oriented MVC pattern. From your question, there is an implied assumption that you don't have a canonical representation for each data resource; approaching Rails this way often increases complexity. If that complexity is truly necessary to meet your application's requirements then that's fine, but I'd certainly recommend carefully assessing your fundamental design goals to see if the functional trade-offs and long-term maintenance burdens are worth it.
I've found questions similar to yours on Stack Overflow; there doesn't appear to be an API or style anyone mentions for persisting across requests. The best you can do seems to be storage in classes or some iteration on what you're already doing:
1) Persistence in memory between sessions/requests
2) Coping with request persistence design-wise
3) Using class caching
I need to create a product configurator, but according to the requirements, literally every product has a set of rules to validate it. The rules refer to the quantity of underlying components the configuration is made of.
At the moment the way this is being handled is just storing the "formula" string in the db, and since the UI is in Excel, then when you call a configuration, it comes with the rules as well and you just append a "=" in front of it. Thus, the final product works when quantities or components change.
So I've seen a few similar type of questions being asked, and the answer always seemed to be UJS, however, this is stored in the app itself, correct? The challenge for me is to create a way I can replicate these rules depending on the product, and different products are beeing added all the time, changed, etc, so keeping it in the app to redeploy each time you want to change something seems a bit extreme!
Can anyone think of a good solution? Help!
These are business rules so they'll need to be stored somewhere server-side (they could be mirrored in client-side code but storing them there exclusively is highly inadvisable ;). They could be represented as code, or configuration files, or in a database.
As you suggested, modeling frequently changing rules in source code makes your app brittle. I think the best storage option is your database.
If you need to implement a client-side behavior on a per-product basis, you could use AJAX to send a product ID out to a service which returns a configuration "package" to your (dumb) client.
Would that work? Sounds good to me, anyway. ;)
I want to be able to define everything about a form field in one place, as opposed to having some info in the DB, some in HTML, some in JavaScript, some in ASP...
Why do I have to worry about possibly changing things in four separate places (or more) when I want to change something about one field?
I.e., I don't want to:
declare the field in the DB
and duplicate some of that info in HTML somewhere
and duplicate some more info in some JavaScript somewhere
and duplicate some more info in some ASP somewhere
Since I'm a developer, I'm ideally looking for a methodology, not a tool or S/W package. (I think!)
Currently, I'm doing this by putting all control information into SQL's extended property "Description" text area. E.g., a required phone number field would have the following SQL declaration:
[home_phone] [varchar](15) NOT NULL
and I put the following "controls" in the Description extended property:
["Home Phone"][phone_text][user_edit][required][allow_na][form_field_size_equals_size][default=""][group="home_address"][rollover="enter only: numbers, dash, parenthesis, space"][explanation="enter <strong>n/a</strong> if you don't have a home phone"]
With my current system, the following HTML is dynamically generated for the Home Phone field:
<div class="div-item" id="item-FORM:FIELD:TABLE_HOME:HOME_PHONE">
<div class="div-item-description" id="item_description-FORM:FIELD:TABLE_HOME:HOME_PHONE">
<span class="rollover-explanation" title="enter only: numbers, dash, parenthesis, space">
<label for="FORM:FIELD:TABLE_HOME:HOME_PHONE" id="item_label-FORM:FIELD:TABLE_HOME:HOME_PHONE">
Home Phone
</label>
</span>
</div>
<div class="div-item-stipulation" id="item_stipulation-FORM:FIELD:TABLE_HOME:HOME_PHONE">
<span class="stipulation-required" id="item_stipulation_span-FORM:FIELD:TABLE_HOME:HOME_PHONE" title="required" >
*
</span>
</div>
<div class="div-item-value" id="item_value-FORM:FIELD:TABLE_HOME:HOME_PHONE">
<div class="individual-forms">
<form class="individual-forms" id="FORM:TABLE_HOME:HOME_PHONE" name="FORM:TABLE_HOME:HOME_PHONE" action="" method="post" enctype="multipart/form-data" onsubmit="return(false);">
<div class="individual-forms-element">
<input
class=""
type="text"
id="FORM:FIELD:TABLE_HOME:HOME_PHONE" name="FORM:FIELD:TABLE_HOME:HOME_PHONE"
size="15" maxlength="15"
value=""
FORM_control="true"
FORM_control_name="Home Phone"
FORM_control_is_required="true"
FORM_control_is_phone_text="true"
>
</div>
</form>
</div>
</div>
<span class="spanExplanation">
enter <strong>n/a</strong> if you don't have a home phone
</span>
</div>
which looks like this (in IE 7):
Client-side JavaScript validation is controlled by the **FORM_control**... parameters, which on error produces explanations and field highlighting. (Unfortunately, custom parameters in HTML elements isn't exactly standards compliant.)
My primary problem is that this method using the Description field has always been cumbersome to use and maintain. The Description property can only be 255 chars, so I have lots of abbreviations. As the system has expanded, the number of controls has also greatly expanded past the original dozen or so. And my code for interpreting all these controls and their abbreviations is just not pretty or efficient. And as I said, custom parameters in HTML elements don't work in FireFox.
Things I'm currently controlling (and want to continue to control) include:
Field description (e.g. "Home Phone Number")
DB table name (e.g., "home_address")
DB field name (e.g., "home_phone")
DB field type/size
DB allow null
Grouping (e.g., this particular field is part of all "Home" fields)
Required/optional
Read-only (for system supplied data)
Size (presented form field size)
Type (e.g., text, numeric, alpha, select, zipcode, phone, street address, name, date, etc)
Accepted input (non-blank; numeric only; no spaces; phone number; reg exp; etc)
Extended explanation (e.g., for phone # "enter n/a if you don't have a home phone")
Roll-over explanation (e.g., for phone # "enter only: numbers, dash, parenthesis, space")
Rows (for select lists -- 1 = drop-down)
Rows/Columns (for textareas)
Error message text
Error indication (how to show which field contains an error, e.g., red background)
Etc...
And to be clear, I'm all for separation of logic and design elements. I do have a separate CSS file which is manually maintained (not part of the generation process).
My server environment is classic (non-.Net) ASP and SQL 2008. I'm pretty good with HTML, CSS, JavaScript, and ASP and I'm comfortable with SQL.
What I imagine that I want is some sort of JSON, XML, etc that is the single source used to generate everything, e.g.:
a SQL script that actually creates the SQL tables
the HTML (with CSS classes and JavaScript client-side validation/function calls)
the ASP (server-side validation)
My current method that does this is dynamic (not compiled) and pretty slow, so I'm probably looking for some sort of "compiler" that generates this stuff once. And I really only have classic ASP, JavaScript or SQL as the available languages for this "compiler".
And while I think I could create this system myself, I'm hoping that other, better developers have already come up with something similar.
Assume this should scale to at least scores of fields. (FYI, my current form has way too many fields on one page, but I'm solving that issue separately.)
Thanks for any help!
Javascript validation is overrated
I think javascript validation is overrated. It was good in the days when a server round-trip could take 10's of seconds but typically now it can take less than 3 seconds. In you factor in an AJAX submission process you can bring that time down to sub-second.
In return for all that effort to slice off a round-trip you have to deal with all the various complexities of cross-browser support, complex debugging, lack of server-side logging and dealing with the case where JS is disabled by the user. In a typical scenario we're talking about a lot of wasted hours and difficult debugging (try asking a typical idiot what browser they use, let alone what version they're using).
The database as a one-stop validator
You said the database isn't a complete validation environment but I think that's no longer true. A modern database like PostgreSQL will allow you to hook up complex validation functions as triggers in pretty much your language of choice and return appropriate error responses to the application.
So if you follow where I'm going here it IS possible to validate in one place, the database, without the historical drawbacks. The process is:
Create a basic HTML form, forget
HTML5 or Javascript validation.
When the form is complete, or as
required, submit it via AJAX (if
available) or standard POST if not.
Pass the UPDATE/INSERT more or less
straight to the DB where your
trigger functions normalise and
validate the data.
Immediately return the result and or
errors (probably via a transaction),
and perform any further server
processing at this stage. If you
decide not to keep the data you
could either delete the new row or
rollback a transaction.
On conclusion return any appropriate
redirection, messages or updates to
the browser via JSON/AJAX or a
reload with the cleaned data.
This may sound slow/inefficient but I think that's ignoring todays realities, namely:
Pretty much everything is broadband now, even wireless.
Processing power is cheaper than
developer time.
These sorts of
updates tend to be limited by the
speed users can fill out forms, you're not going to
hammer your DB in a typical
scenario.
You still have to do the validation somewhere, so why not the DB?
And the benefits are huge:
On a high volume server (like an exchange, twitter, feed, etc) this
process lends itself to API control
via SOAP/AJAX/RSS/whatever since
only a thin layer is need to
transfer data between the API client
and the DB.
No matter what client
language or protocols are used the
validation remains the same.
Even a raw SQL statement gets
validated which can prevent
programming errors, corrupted
imports or 3rd-party feeds from
destroying your data structures.
Triggers are easily toggled if
required. It can often be harder in normal code.
Validation is always consistent.
Validation functions live inside the
database, allowing then to access
indexes and other rows or tables without
connector overhead, data conversion
and network/socket lag.
Validation functions can run in
compiled code, even if your web
server language is dynamic.
The only real drawbacks are:
Difficult to upgrade or migrate to
other database software.
Difficult if your preferred language
isn't supported (However Postgres
supports functions written in C,
PL/pgSQL, Python, TCL, Perl, Java,
R, Ruby, Scheme, Bash and PHP so
unless you're stuck on C#/VB you
should find one you can handle).
Context sensitivity
There are some aspects of your question I wouldn't recommend at all. Primarily where you're trying to tie the presentation of HTML form objects to your data in a single location.
This idea is going to backfire very quickly because you will find that in a typical application the presentation of information is highly sensitive to context - specifically the target audience.
For example on an ordering system you may have data entered by a client that is then accessible to an admin. The clients view of the data may be more limited, it may need a different title and description, it may be preferable to display as checkboxes while a admin gets a more compact view. You may even be presenting the same data to different types of client (retail vs. wholesale).
In short, the presentation of data is typically required to be more fluid than its validation so at some point you should really draw the line - even if that means some repetition.
I've been working on exactly the same problem at my job. I can't stand repeating myself, particularly because I know that when I have to change something months later, I'll never remember all of the scattered redundant pieces. The answer must take into account the following truths:
The database should, as much as is reasonable possible, validate itself. This is basic data integrity; the DB should throw a fit if you try to put invalid data in it.
The database cannot validate itself. It's easy to add constraints for uniqueness, or format, or foreign keys, and technically SQL can go a lot further, but if you're enforcing, say, address/zip code correspondence at the database level, you're going to regret it. Some part of the validation logic must live in the server-side code. In your case and mine, this means the ASP.
If you want client-side validation, this means Javascript.
At this point, we're already talking about validation constraints in three languages, and the impedance mismatch between them may be significant. You can't always factor validation out to one of them. All you can do is keep the logic together as much as possible.
The solution you suggest has one giant advantage – that all of the logic is in one place, together. This advantage is balanced by several drawbacks:
You can't do any validation at all without talking to the database.
In order to get the metadata from the database to your ASP, you have to have special code to interpret your metadata minilanguage. This is far more complex than accepting some degree of redundancy.
Your metadata puts front-end display code in your database. This means that if you want to change the text, you have to edit your database model. This is a rather drastic requirement which ties your database model to your presentational logic. In addition, internationalization is virtually impossible.
Because there are so many translational layers between your metadata and the user, any extension to your metadata space will require the revision of several layers of tightly coupled code.
To try to find a middle ground between your solution and the redundancy it's designed to avoid, I suggest the following:
Put basic validation constraints in the database.
Create a system in ASP for specifying a behavioral data model with arbitrary contents and validation constraints. When you define your model using this syntax, you will duplicate only the bare-bones constraints in the database.
Create a system in ASP to display form fields on the page in HTML. A form field declaration will reference the appropriate data model and additionally include display code such as labels and descriptive text. The HTML generation code can use sensible defaults derived from the data model. The only duplicated data should be the name of the field, which is used as a key to bind a displayed field to the appropriate data model.
Create or find Javascript validation library. Have the aforementioned HTML generation code automatically insert hooks to this library in the generated markup based on the associated data model.
Thus, you have a system where information about a field may be stored in a handful of places, depending on where it is most appropriate, but almost never duplicated. Validation information is declared in the ASP data model. Display information is found only in the on-page field declaration. The field name is used throughout this stack as a key to link them together, and the hierarchy of concerns allows you to override assumptions made on lower levels as needed.
I'm still working on my implementation of this design, but if you're interested, I can post some sample code.
Seems to me that this gos against every principle of separation of logic and design elements. I know that on the bigger projects I worked on, there were actual SDLC requirements that dictated one type of engineer could touch one level of file, while a UI Engineer could touch another and a "code monkey" could only touch a subset of that. Could you imagine the chaos that would ensue in that scenario? The Code monkey would have to get permission from the UI Engineer who in turn would have to coordinate with the engineer who would have to join on a conference call with integration who would have to ping tech support who then would shelve the project until business asked legal....
All kidding aside, I don't think your method is bad.
I do believe in handling things as they were meant to be handled natively, i.e. building a form text field is likely more efficiently handled by html natively than by database calls which then build html via a series of scripts. Your "compiled" method makes me wonder if it would cancel out the benefits of caching of common javascript and css elements in their respective files.
There are frameworks such as Zend, CodeIgniter, and Symfony (on the PHP side) that are getting closer to what you mention via built-in functionality....although they're not there yet. Zend in particular uses programmic features to build, validate and style forms, and once you figure out its nuances it's quite powerful. Perhaps it could serve as a model for your ultimate quest. Although it seems like you're a classic asp guy....and that's not what you're looking for. I digress.
I think this question is out of my scope of knowledge, but I figure it doesn't hurt to try and help.
I'm working on my first PHP site and since the beginning, since I could not predict many factors of the site and only being one person, I decided from the beginning that every design element on every page will be maintainable through one page. This is a learning experience, so I'm not too concerned that there wasn't much planning involved, but some things just grind my gears, like naming conventions, but with my method, I always am able to make site wide changes with ease.
Every page I make is structured like this:
<?php require_once 'constants.php'; ?>
<?php $pageTitle = 'Home Page'; ?>
<?php require_once REF_LAYOUT_TOP; ?>
<h1>Hello!</h1>
<p>World</p>
<?php require_once REF_LAYOUT_BOTTOM; ?>
In the constants, I have constants for just about everything. CSS colors (for consistent layout), directory locations, database connections, links, constants just for certain pages (so I can modify file names and not have them damage anything), and all sorts of things.
The top part contains navigation, error handling JavaScript scripts, any kind of dynamically created content, the navigation, etc.
This way, if I ever want to implement something new, it's implemented every where. I gave jQuery a shot and it required only one link.
Possible Solution
If you are trying to adjust many things from one location, I highly suggest you invest in a little PHP knowledge. Since PHP is just a server script, it's only output is text. In other words, you can insert PHP in JavaScript, HTML, and just about anywhere. This is how you can set up the same text for all sorts of hover pop ups. I don't know if ASP will prevent you from doing this (I have zero knowledge of it).
I figure this is how most sites must be constructed. It has to be ... how else could they maintain hundreds of pages? I think this is the most logical and semantically correct.
I'm not familiar with ASP, so I'll be speaking more generally without knowing how they're implemented.
Generally, a form represents the information needed to create, edit, or delete an entity. So I'd start with an Entity class. In other architectures this is typically called a model (as in Model-View-Controller). The Entity class determines what information it needs, and it takes care of the database queries.
A form could be built from the Entity directly this way. The Entity gives you more direct control, for example, you may have a field in the database for an integer, though the value you really want is between 0 and 255. The Entity can know this more specific constraint, even if the database doesn't.
Next, you could create some sort of form class that would use an entity to generate its interface. It would take care of all the HTML, Javascript, and whatever else you needed.
The entity could have a good variety of types. The representation in the database can be effectively separated. Let's say a post can have many tags. In the database you'd probably keep two tables, one for posts and another for tags. But the entity would represent a post, along with a list of tags, so they're not separate.
The form class can take care of what this looks like, and you just worry about the semantics. For example, if the entity calls for a list of strings, the form could implement that by using Javascript to create an expanding list of text fields, then the form takes care of properly submitting this data to the Entity.
The form would also make a difference in multiple fields working together, or parsing. For example, if the form saw a type that could be null, it would offer an explanation saying "Type n/a if you don't have a phone number" and if it saw that string, correctly return null.
A Type class could be an interface to validate form data. If all the validate() methods on all the types return true, the form is submitted. Each type also takes care of parsing its values (like the "n/a" parsing) so the right thing is submitted.
One point of this is that forms are not analogous to tables. An id field in a table shouldn't show up in a form, and some data may be connected to it in another table, so think of the forms in terms of the "entity" it's modelling, not the table. It's just an adaptor.
I define my schema for each table in XML files. Then I wrote one set of CRUD methods that can operate on any XML schema (passed in as a request paramter). Beside CRUD it can create and drop the table, export the contents to CSV and import a CSV file as well. All I have to do is drop an new schema file in my schema directory and I have full CRUD for this new table. If a field is a FK, a link automatically appears next to the input box during an INSERT or UPDATE that when clicked, pops open a window to look up the foreign key. If the field is a DATE, a link for a pop up calendar appears automatically.
I did this using Java EE and jsp. But I'm sure it could be done with php as well.
<schema>
<tableName>xtblPersonnel</tableName>
<tableTitle>Personnel</tableTitle>
<tableConstraints></tableConstraints>
<!-- COLUMNS ====================================== -->
<column>
<name>PID</name>
<type>VARCHAR2</type>
<size>9</size>
<label>Badge ID</label>
</column>
<column>
<name>PCLASS</name>
<type>VARCHAR2</type>
<size>329</size>
<label>Classification</label>
</column>
<column>
<name>PFOREMAN</name>
<type>VARCHAR2</type>
<size>9</size>
<label>Foreman Badge</label>
</column>
<column>
<name>REGDATE</name>
<type>DATE</type>
<size>10</size>
<label>Registration Date</label>
</column>
<column>
<name>PISEDITOR</name>
<type>VARCHAR2</type>
<size>3</size>
<label>Is Editor?</label>
<help>0=No</help>
<help>1=Yes</help>
</column>
<column>
<name>PHOME</name>
<type>VARCHAR2</type>
<size>9</size>
<label>Home?</label>
</column>
<column>
<name>PNOTE</name>
<type>VARCHAR2</type>
<size>35</size>
<label>Employee Notes</label>
</column>
<!-- Primary Keys ====================================== -->
<!-- The Primary Key type can be timestamp, enter, or a sequence name) -->
<primaryKey>
<name>PID</name>
<type>enter</type>
</primaryKey>
<!-- FOREIGN KEYS ====================================== -->
<!-- The Foreign Key table is the lookup table using the Key to retreive the Label -->
<foreignKey>
<name>PID</name>
<table>phonebook</table>
<key>badge</key>
<label>lname</label>
</foreignKey>
</schema>
Bravo! Your idea is pretty good and the concept is in the right direction, and it already has been done by multiple companies. That was the original "RAD" (Rapid Application Development) concept.
The idea was to keep the attributes of each field in a database, aka "metadata repository" or "data dictionary". This is not only a good idea, but it is a best practice, so all the fields are consistent in type, length, description, etc. The data dictionary should be used not only with the user interface, but also with the database creation. Going a bit further, with this approach you can easily handle multiple locales.
Unfortunately, RAD tools are not that common these days. They are expensive, and in some cases inflexible and restrictive. Programmers love to program, and look with disdain those kind of tools. But, who knows? A new open source project seems to start every day!
Unfortunately your "tool set" is pretty limited, and creating a RAD tool is no trivial task: it involves an unexpected degree of complexity. You probably need to learn .NET, Java, or any other powerful language.
The best approach is to create a tool that, based in your data dictionary stored in a database, generates the ASP or whatever HTML required, so you improve performance. If the dictionary or a form change, you simply run your generator, and voila!, your new page is ready.
You also need to be able to allow "overriding" the dictionary if required. For example, in some cases the word "Telephone" will be too long for certain forms. Furthermore, you also need to have a code generator good enough, so you don't have to manually modify your generated code, and if you need to do it, your tool will be smart enough to remember those changes.
Unfortunately I can't help you more with this. My recommendations are: (1) improve your skills, (2) look for open source projects that do what you need, (3) if willing, help the project, and (4) leave everybody in the dust generating applications faster than anybody else. ;)