How to dynamically generate Business Connectivity Services - sharepoint-2010

Question

We are trying to deploy a Business Connectivity Services(BCS) Model solution where the properties in the model depend on the structure of the data exposed by a web service.
Ideally the BCS Model would expose a collection of key/value pairs which are then converted to columns in a sharepoint list later as this would mean that the same model could be used for several different datasets, however from what we can tell this is not how BCS Models have been designed, as they rely on the model to be strongly typed in order to reflect the entity being imported.
As it stands we are therefore looking at a solution which enables the user to "create" a new external list by providing the url to a remote dataset through a custom page in sharepoint central admin, that will then automatically construct the BCS Model project (by altering a project template) and then compiling and releasing the resulting feature on the fly.
This way we can create the "fixed" class with properties that represent the structure of the data being imported.
for example, datasource A could expose
<cars>
<car>
<color>blue</color>
<make>ford</make>
</car>
<car>
<color>red</color>
<make>lotus</make>
</car>
</cars>
in which case we need a BCS Model for "car" which has two public properties , color and make
however datasource B could expose
<invoices>
<invoice>
<amount>£34.00</amount>
</invoice>
<invoice>
<amount>£34.00</amount>
</invoice>
</invoices>
in which case we need a BCS Model "invoice" with a single public property for amount.
Would appreciate anyones feedback on this approach or the 'best practice' way of achieving this.

Answer 1

[I've had experience doing something similar in .Net - I'm not sure how relevant this will be to you.]
I had to write an import tool that could handle any file format. To handle this properly, I wrote a small class which would take an xml format definition (name, data type, format string, custom parser, etc..) and generate a class which could read the file and expose an IQueryable<FileFormat> and some additional metadata.
It's worth noting that to make it completely flexible, I had to allow the format definition to provide a C#/VB lambda which would be compiled and executed (eg, when the input date format was non-standard and needed a custom parser). This is clearly a security risk - so when I instantiated the dynamic class, I did so in a seperate AppDomain with very few privilieges. This may not apply in your situation.
We used a custom templating engine to generate the code which was then compiled using the System.Codedom.Compiler namespace - This allowed us to create assemblies and cache them until the definition changed. It may be worth considering the Razor templating engine if you do something similar.
The only real issues we had arose from coding against an unknown data type. By making the custom class implement our own IImportFile interface which exposed metadata in a standard way (effectively the same info as in the xml spec), we could work around it without too much effort.
We were lucky in that this is a tool for trusted users (at least only trusted users can provide a new file format specification) so the security risks were limited. If you're compiling code based on user input, make sure you've got adequate safeguards in place.