This is not a code/case specific question.
I am new to Apex, and I'm trying to test methods that do Callouts to external APIs. I understand that in order to test this method, I have to create a class that implements HttpCalloutMock and use it in my test.
However, I want to know: in the Test, when I call the actual method I'm testing, does a call go out to the API behind the scenes? Or is the data I'm putting in the mock the only data that gets passed around?
(I'm asking because, if the latter, wouldn't that mean these tests are extremely counterproductive and unnecessary?)
The dummy data you provided in the mock class will be dutifully returned. And yes, it's annoying, double work.
But how else could it be done? Really calling an external API might have bad consequences (sending "My Awesome Test Order!!!1one!eleven" to production fulfilment system would be a disaster, especially if you do it few times because deployment kept failing). And when such API would be down and you really, really need to deploy something to production - you shouldn't be a hostage of 3rd party server, even test one.
Instead of grumbling try to embrace it. Yes, it's rubbish. But this is your opportunity to test how your code handles different outputs. How it reacts when the API response is "HTTP 500 Internal Server Error", HTML instead of JSON or even there's no response, just timeout. The more solid you make it, the more confident you'll be.
Is it really that hard? Capture couple real messages & errors, remove sensitive data, implement some switch statement "if account number = 123 return this else return that" and you're done.
And yes, it essentially means implementing 3rd party's logic yourself. But well, with test-driven development you ideally would start with a dummy representation of their service anyway, something that's close enough to the API "contract" you have. And as a bonus - you get to shout at them when something suddenly breaks and you can prove it wasn't a change on your end.
In the end it's not too different from splitting work with another SF developer. "OK, I'll do the UI bit, you do the apex bit, here's the data interface we promise to use, see you in 1 week's time". How far can you trust the guy, eh? ;)
Related
I know that GET is used to retrieve data from the server without modifying anything. Whereas POST is used to add data. I won't get into PUT/PATCH, and assume that POST is always used to update and replace data.
The theory is nice, but in practice I have encountered many situations where my GET calls need to be replaced with POST calls. This is because the response often gets incorrectly cached. Where I work there are proxy servers for security, caching, load balancing, etc., and often times the response for GET calls is directly cached to speed up the call, whereas POST calls never get fully cached.
So for my question, if I have an API call /api/get_orders/month. Theoretically, this should be a GET call, however, the number of orders might update any second. So if I call this API at any moment it may return for example 1000, and calling it just two seconds later should return 1001. However, because of the cache, and although adding a version flag such as ?v=<date_as_int> should ensure that the updated value is returned, there seems to be some caches in the proxy servers that might ignore this.
Basically, I don't feel safe enough using GET unless I know for certain that the data will not be modified or if I know for a fact that the response is always the updated data.
So, would you recommend using POST/GET in the case of retrieving daily/monthly number of orders. And if GET, with all the different and complex layers and server set-ups, how can one be certain that the data is always updated?
If you're doing multiple GET request and something is caching the data in between, you have no idea what it is or how to change it's behavior then POST is a valid workaround.
In any normal situation you would take the time what sits in between your browser and your server, and if there's something that's behaving in a way that doesn't make sense, I would try to investigate and fix that.
So you work at a place where some of that infrastructure exists. Maybe talk to the people that maintain it? But if that's not an option and you just want to find the 'ignore every convention and make my request work'-workaround, then you can use POST.
I'm looking for assistance for the bare minimum code to pull some information from the TheTVDB API (v3).
I've never coded anything to do with APIs before.
I tried to shortcut using TVDBSharper, but that uses asynchronous routines, and tasks, etc. which I just can't get my head around at the moment, given the documentation is for C#, and I clearly don't understand how "await" works in VB.
I've tried searching for API examples, but most are about creating an API.
The first thing TheTVDB API documentation says is:
"Users must POST to the /login route with their API key and credentials in the following format in order to obtain a JWT token."
^ I don't know how to POST. Any examples I've seen are very long and confusing, and mostly in C#.
So (and I apologise for this drivel, but I've tried on and off for months now)…
Could someone please show me the minimal amount of VB.NET code to pull the show name from, for example series ID 73739 (Lost). Hopefully from there, I can start to figure some things out.
I have a valid API Key from the TheTVDB.
Mostly you don't need to understand async/await in any great detail but I was once where you are now, and though I don't claim to be an expert, I did manage to get my head around it like this:
You know how, if you had something that threw an exception and you never caught it:
Sub Main(arguments)
Whatever()
End Sub
Sub Whatever
StuffBefore()
OtherWhateverThrowsException()
StuffAfter()
End Sub
Sub OtherWhateverThrowsException()
StuffBefore()
throw New Exception("Blah")
End Sub
As soon as you threw that exception, your VB thread would stop what it was doing, and wind its way back up through the call stack until it popped out of the main, and crashed to the command line - a matrixy style "return to the source" if you like
Async Await is a bit like that. When there's some method that is going to take a long time to do its work (download strings from tvdb) we could make it sit around doing nothing in our code, having a up of coffee and waiting for TVDB's slow server. This makes things easy to understand because if we sit and wait, we wait 30 seconds, then we get the response, and process the response. Obviously we can't process the response before we get it so we have to sit around and wait for it, and this is always true..
But it'd be better if we could let our thread nip back out the way it came in, "go back to the source", do something else for someone else, and then call it(or another one of its coworkers, we probably don't care) back to carry on working for us when TVDB's server responds. This is what Async Await does for us. Methods that are marked Async are treated differently by the compiler, something like saving your progress on your xbox game. If you reach a point where you want to wait, you can issue the waiting command, the thread that was doing our work performs a savegame, goes off and works for someone else, then when we're ready it comes back, loads the game again and carries on where it left off.
The save game file is manifest as a Task; methods that once upon a time were subs (didn't return anything) should now be Functions that return a Task (a savegame with no associated data). Methods that once upon a time returned something like a string, should now be marked as returning a Task(Of String) - the Task part is to save the state of play (data that VB wants to work with), the string is the data your app wants to work with.
Once you mark something as Async, it needs to contain an Await statement. Await is that SaveYourGameAndGoDoSomethingElseWhileThisFinishes. Typically, while you're awaiting something your program won't have any other stuff it needs the thread to do, so it's not just your Function that calls TVDB's API that needs to Await/be marked Async - every single function in the chain, all the way up and out of your code, needs to be marked as Async, and typically you'll Await at every step of the way back up:
Sub DownloadTVDBButton_Click(arguments)
DoStuff()
End Sub
Sub DoStuff
StuffBefore()
GetFromTVDB()
StuffAfter()
End Sub
Sub GetFromTVDB()
Dim i = 1
GetDataFromTVDBServer() 'wait 30s for TVDB
ParseDataFromTVDB()
End Sub
Sub ParseDataFromTVDB()
End Sub
Becomes:
Async Sub DownloadTVDBButton_Click(arguments) 'windows forms event handlers are always subs. Do not use async subs in your own code
Await DoStuff()
End Sub
Function DoStuffAsync Returns Task
StuffBefore()
Await GetFromTVDBAsync()
StuffAfter()
End Function
Async GetFromTVDBAsync() Returns Task
Dim i = 1
Await GetDataFromTVDBServerAsync() 'go back up, and do something else for 30s
ParseDataFromTVDB()
End Sub
Sub ParseDataFromTVDB() 'downstream; doesn't need to be async/await
End Sub
We switched to using TVB's Async data call, so we await it. When we await, the thread would go back up to the previous function DoStuffAsync. Because we're awaiting that, the thread goes back up a level again into the button click handler. Because we're awaiting that also, the thread goes back up again and out of your code. It goes back to its regular day job of drawing the UI, making it looks like the program is still responding etc. When the TVDB call completes the thread comes back to the point just after it (ready to run ParseData), and it has all the data back from TVDB, and the savegame has been reloaded so everything it knew before/the state is as it was (variable i exists and is 1; you could conceive that it would have been lost otherwise when the thread went off to do something else)
In essence, async/await has allowed us to work exactly as we would have done without it, it's just that it built a little savegame mechanism that meant our thread could go off an do something else while TVDB was busy getting our data, rather than having to sit aorund doing nothing while we waited
It may also help to think of Await as a device that unpacks a save game and gets your data out of it. If a GetSomething() sits for 30s then returns a String you want, then GetSomethingAsync() will instantly return a Task that will (in 30s when the work is done) encloses that same String you want, and Await GetSomethingAsync() will wait until the Task is done then get the string you want out of it
Methods that are named like "...Async" should be thought of as "behave in an asyncronous way". They DON'T have to be marked with the Async modifier; Async is only needed if a method uses the Await word but I'm recommending you use Await on everything that returns a Task (i.e. is awaitable) all the way up and down your call tree. When you get more confident you don't always have to Await SomethingAsync but honestly the overhead of doing so is minimal and the consequences of not doing so are occasionally disastrous. All developers who follow convention always name their stuff ...Async if it behaves in an async way; you should adopt this too, and make sure you name all your Async methods with an"Async" at the end of the name
I don't know how to POST
You don't really need to. The TVDB API has a swagger endpoint; swagger is a way of describing a REST service programmatically so that your visual studio can build a set of classes to use it and provide you with nicely named things. Whipping out a WebClient and manually creating some JSON is very old school/low level
TVDB's swagger descriptor is at https://api.thetvdb.com/swagger.json
You're supposed to be able to right click your project, choose Add... Rest API Client:
,
Paste https://api.thetvdb.com/swagger.json in as the url and pick a namespace (an organizational unit) for all the generated classes to go in.
At the moment something in TVDB's API is causing AutoRest (the tool that VS uses to parse the API spec) to choke but ordinarily it would work out and you'd get a bunch of code (autorest generates c#; you'd be best off generating the c# into a new project and then adding reference to that project from your VB) objects to work with that would do all the POSTing etc for you.
As noted, my VS can't process the TVDB API at the moment and I dont have enough time today to figure out why, but you could sure post a question on AutoRest's github (or on SO) saying "why does https://api.thetvdb.com/swagger.json cause a "Input string not in correct format"" and get some more help
You asked (maybe implicitly) a couple of follow up questions in the comments:
I don't know about REST/swagger (I've heard of it though), and can't see any way to add to the project as you described, and I'm no closer to getting info from TheTVDB. However, it might have have helped me use functions in TVDBSharper. I will just have to try a few things with it. Thanks again
Yes; sorry - I should have been more explicit that "Add REST API client" is only available in a C# project because it relies on a tool that generates C#. This isnt a blocker though - you can just make a C# project and add it to your VB solution alongside your VB project; the two languages are totally interoperable. Your VB can tell your C# what to do
However, there isn't much point in trying at the moment, because the tool that is suppsoed to do it can't handle what TVDB is putting out; my VS can successfully ask the TVDB API to describe itself, but it doesn't seem able to understand the response.
In a nutshell; VS has a bug that means it can't use TVDB API directly, you're best off trying via TvDbSharper. The https://github.com/HristoKolev/TvDbSharper readme has some examples in. They're C# but basically "remove the semicolons and they'll pretty much work in VB"
Now, a bit about the headline terms here, background understanding if you like. API, RESTand swagger are easy enough to explain:
API
An API is effectively a website (in this case run by TVDB), intended for software to consume rather than humans. It takes raw data in and chucks raw data out - unlike a normal website intended for our eyes, nothing about it is presentational in the slightest.
REST
REST as a phrase and a concept is a source of confusion for many and a lot of times you try and read about what REST means and the blogs quickly start getting bogged down with details and make it too complex, with all these funky examples. They kinda forget to explain the REST part because it's come to mean not much at all - it's something so obvious and nondescript that we don't think about it any more.
In essence, something is RESTful if the server doesn't have to remember something about what you did before, in order to service a request you make now - every request stands on its own and can be serviced completely without reference to something else. This is a different workflow to other forms where you might want to change the name of something by issuing a editname('newname') command. What name actually gets edited depends on whether you first did selectshow() or selectactor() and also which show or which actor - a workflow like that means the server has to start remembering whether you selected a show or actor, and what show/actor was selected before it can process the editname() command. If you selected show 123, the edit would edit the name of the show id 123. If you selected an actor 456, the edit name would edit the name of an actor 456
Critically, if you replayed the same editname() at a different time a different thing would get edited because the state of your dialog with the server changes. It's kinda dumb to make the server have to remember all that, for everyone, when really we could push the job of identifying whether we want to name an actor or a show and which show, onto the client
By making it that you have editactorname(123,'Jon wayne') you're transferring all the info the server needs to perform the request; your credentials, the actor id, the new name, the fact that it's an actor name and not a show name. All this goes in the one request, and you can replay this request as many times as you like at any time, and it always has the same effect; things that happened before don't affect it (well.. apart from authentication)
It gets a bit woolly if taken literally - "well if the server doesn't remember anything how does it even remember I changed the name of actor 123, to Jon Wayne so it can service my later request of getactorname(123)?" but that's more about the state of the data in the server, not the state of your interaction with the server. Things that are truly stateless are mostly purely calculatory and not too useful; something somewhere needs to be able to remember something or there is nothing to calculate. Things are rarely completely stateless; even TVDB's API requires you to authenticate first, using a user/password/apikey and then the serverissues a token that becomes your username/password/apikey equivalent for every subsequent request - the server has to start remembering that token, or every time you quote it it will say "can't edit actor name; not authorized". So, yeah.. when viewed holistically something usually has to be rememberd at some point otherwise nothing works. REST things are rarely 100% truly stateless, but mostly they are - and it's really about that "when you want to edit the actor name, send a) that you want to edit actorname, b) what actor, c) what name, d) your credentials to prove youre allowed to" - everything the server needs in the one hit
Swagger
Now called OpenAPI, swagger is a protocol for describing an API: when an api has some actions that take some data, and return some data, it's helpful to know what the actions are called (setactoryearsactive), what type of data they take (date, date), what sort of things you should put in it (the from date, the to date or null if still active), what they return (boolean) and what the return means (true if success, false if not).
If we have a standardized way of describing these things then we can build standard software that reads the standard description of the API and writes a bunch of standard code that uses the API. This is software that writes a description so other software can read it and write software that uses the first set of software. It's an API API.
There is a lot of software here:
The API is software(tvdb),
The thing that generates the description of the API is software (Swagger),
The thing that consumes the description of the API and creates a client is software(AutoRest),
And the thing that uses the client is software (your app).
You could code your app to hit the api directly- the API's just responding
to HTTP requests, which are just text files formatted in a particular way sent to port 80 of the web server that hosts the API. You could write one such request in notepad and use telnet to send it and get a valid response. You could code your app to do it (you were just about to). You could use someone else's library (TvBbSharper) which does it somehow. You could use some software that generates something like TvDbSharper; it reads the description of the api and generates classes for you to use; those classes will make the http requests. Everything can be done at any level; you could write all your apps in assembler, the lowest of the low. It takes ages and it is boring - this is why we use ever higher levels of abstraction.
We make something and then make it do a thousand things and then realize that listing the same code over and over and changing one bit each time is boring, and repetitive and something a computer should do, so we devise ways of making it so software can write the boring repetitive code so that we can do the interesting things.
Swagger and AutoRest are those kind of things; Swagger inspects all the methods, what they take and return and generates a regular consistent description. AutoRest reads it and generates a regular consistent set of client classes. Then the human uses the client classes to do the interesting things. The AutoRest part doesn't work out for us at the moment; it's written by different people than the Swagger team so some differences arise; Awagger describes something and Autorest can't understand it. It will one day I'm sure (in this game of walls and ladders); such is the nature of open source - everyone has a different set of priorities.
Right now we could probably get AutoRest working by finding the one thing it is choking on and removing it. There may be no need; if the TvDbSharper guys have written enough of a set of client classes that you can use TvDbSharper to do all your necessary things. It is thus effectively already the set of client classes AutoRest would have built, maybe more; use TvDbSharper.
The idea behind Swagger and Autorest is that a TvDbSharper shouldn't need to exist: it's a very specific application, only works with tvdb, only works in .net.
If we put effort into making Swagger able to generate a description of any API written in any language, and we put effort into making Autorest able to consume that description and output any language, then we have something more useful than TvDbSharper/no need to TvDbSharper because we can generate something that does the same (of course, specific applications can be superior, just like bespoke tailored suits are superior bt that's another philosophy for another time)
I am dealing with a coverage test of an command-line interface application developed in Elixir. The application is a client for tldr-pages and its functioning consists in a script built with escript. To perform the actions I use a case structure over an HTTPoison.get/1 function, in which I introduce the formatted url. In this case I compare the response to different kind of values, such as if the page exists, it shows the information; if the not, it report it to the user and then continue in another case to evaluate to others possibilities. At the end, the first case finish with two pattern to match errors, one for the lack of internet connection and another one for unexpected errors. The described structure is the next one:
case HTTPoison.get(process_url(os, term)) do
{:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
IO.puts(body)
{:ok, %HTTPoison.Response{status_code: 404}} ->
IO.puts(
"Term \"#{term}\" not found on \"#{os}\" pages\nExTldr is looking on \"common\" pages."
)
case HTTPoison.get(process_url("common", term)) do
{:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
IO.puts(body)
{:ok, %HTTPoison.Response{status_code: 404}} ->
IO.puts("Term not found on \"common\" pages.")
end
{:error, %HTTPoison.Error{reason: reason}} when reason == :nxdomain ->
raise NoInternetConnectionError
{:error, %HTTPoison.Error{reason: reason}} when reason != :nxdomain ->
raise UnexpectedError, reason
end
NoInternetConnectionError and UnexpectedError are exceptions defined in another file. Both patterns at the end works apparently nice, at least the first one:
{:error, %HTTPoison.Error{reason: reason}} when reason == :nxdomain ->
raise NoInternetConnectionError
However, as I said at the beginning of the question, I am dealing with a coverage test performed automatically with GitHub Actions and Coveralls with ExCoveralls in the dependencies. In this test I am receiving a warning for both raise/1 statements. Although I could be wrong understanding what means that, I understand this "missed lines" or uncovered lines are reported by Coveralls because I am not performing a test to cover this case. Thus I started to research how I could write a test to cover both conditions.
The most important issue is how to simulate the lack of internet connection in a test to cover this. I though about to develop a mock, but I do not find something useful for this case in some of the mocking packages for Elixir, like Mox. Then I found bypass, a very interesting package that I think it might be useful because it has down/1 and up/1 to close and start a TCP socket, so it makes possible to test what happens when HTTP server is down. But with this I have two issues:
A server down is not the same that the lack of internet connection by the user part.
I tried to applied this "down and up" mechanism and I did not accomplish it. I am not going to share it because I think it would not be the final solution by the first issue described.
I am not pretending an answer with the code that solves this problem, I am just trying to understand how this test should work and the logic I should follow to develop it. I am even researching Erlang documentation, because it is possible Erlang provides native functions to address it (for example, I am now reading the Erlang's Common Test Reference Manual, because may be there is something useful there).
Edit. What I commented I tried with bypass was to install the dependency, write a setup with bypass.open/0 and then write a test like the next one, in which I try to assert the capture output with capture_io/1:
test "lack of internet connection", %{bypass: bypass} do
Bypass.down(bypass)
execute_main = fn ->
ExTldr.main([])
end
assert capture_io(execute_main) =~ "There is not internet connection"
end
However, as I thought, it does not cover the possible situation of lack of internet, just the possibility to check when a server goes down.
Sidenote: I personally was always against being a slave of tools that are supposed to help the development. Coverage is a somewhat good metric, but the recommendations should not be treated as a must. Anyway.
I am not sure why you ruled Mox out. The rule of thumb would be: tests should not involve cross-boundary calls unless absolutely unavoidable. The tests going over the internet are nevertheless flaky: coverage would not tell you that, I would. What if the testing environment has no permanent internet access at all? Temporary connection issues? The remote is down?
So that is exactly why Mox was born. And, luckily enough, HTTPoison is perfectly ready to use Mox as a mocking library because it declares a behaviour for the main operation module, HTTPoison.Base.
All you need would be to make your actual HTTP client an injected dependency. Somewhat along these lines:
#http_client Application.get_env(:my_app, :http_client, HTTPoison)
...
case #http_client.get(process_url(os, term)) do
...
end
In config/test.exs you specify your own :http_client, and voilà—the nifty mocked testing environment is all yours.
Or, you might declare the mock straight ahead:
Mox.defmock(MyApp.HC, for: HTTPoison.Base)
I am also adept of boundaries on the application level calling 3rd-parties. That said, you might define your own behaviour for external HTTP calls you need and your own wrapper implementing this behaviour. That way mocking would be even easier, and you’ll get a benefit of easy changing the real client. HTTPoison is far from being the best client nowadays (it barely supports HTTP2 etc,) and tomorrow you might decide to switch to, say, Mint. It would be drastically easier to accomplish if all the code would be located in the wrapper.
I'd like to write a Minecraft mod which adds a new type of mob. Is that possible? I see that, in Bukkit, EntityType is a predefined enum, which leads me to believe there may not be a way to add a new type of entity. I'm hoping that's wrong.
Yes, you can!
I'd direct you to some tutorials on the Bukkit forums. Specifically:
Creating a Meteor Entity
Modifying the Behavior of a Mob or Entity
Disclaimer: the first is written by me.
You cannot truly add an entirely new mob just via Bukkit. You'd have to use Spout to give it a different skin. However, in the case you simply want a mob, and are content with sharing a skin of another entity, then it can be done.
The idea is injecting the EntityType values via Java's Reflection API. It would look something like this:
public static void load() {
try {
Method a = EntityTypes.class.getDeclaredMethod("a", Class.class, String.class, int.class);
a.setAccessible(true);
a.invoke(a, YourEntityClass.class, "Your identifier, can be anything", id_map);
} catch (Exception e) {
//Insert handling code here
}
}
I think the above is fairly straightforward. We get a handle to the private method, make it public, and invoke its registration method.id_map contains the entity id to map your entity to. 12 is that of a fireball. The mapping can be found in EntityType.class. Note that these ids should not be confused with their packet designations. The two are completely different.
Lastly, you actually need to spawn your entity. MC will continue spawning the default entity, since we haven't removed it from the map. But its just a matter of calling the net.minecraft.server.spawnEntity(your_entity, SpawnReason.CUSTOM).
If you need a skin, I suggest you look into SpoutPlugin. It does require running the Spout client to join to such a server, but the possibilities at that point are literally infinite.
It would only be possible with client-side mods as well, sadly. You could look into Spout, (http://www.spout.org/) which is a client mod which provides an API for server-side plugins to do more on the client, but without doing something client side, this is impossible.
It's not possible to add new entities, but it is possible to edit entity behaviors for example one time, I made it so that you could tame iron golems and they followed you around.
Also you can sort of achieve custom looking human entities by accessing player entities and tweaking network packets
It's expensive as you need to create a player account to achieve this that then gets used to act as a mob. You then spawn a named entity and give it the same behaviour AI as you would with an existing mob. Keep in mind however you will need to write the AI yourself (you could borrow code straight from craftbukkit/bukkit) and you will need to push the movement and events of this mob to players within sight .. As technically speaking all your doing is pushing packets to the client from the serve on what's actually happening but if your outside that push list nothing will happen as other players will see you being knocked around by invisible something :) it's a bit of a mental leap :)
I'm using this concept to create Npc that act as friendly and factional armies. I've also used mobs themselves as friendly entities (if you belong to a dark faction)
I'd like to personally see future server API that can push model instructions to the client for server specific cache as well as the ability to tell a client where to download mob skins ..
It's doable today but I'd have to create a plugin for the client to achieve this which is then back to a game of annoyance especially when mojang push out a new release and all the plugins take forever to rise with its tide
In all honesty this entire ecosystem could be managed more strategically but right now I think it's just really ad hoc product management (speaking as a former product manager of .net I'd love to work on this strategy it would be such a fun gig)
What's the best strategy to use when writing JMeters tests against a web application where the values of certain query-string and post variables are going to change for each run.
Quick, common, example
You go to a Web Page
Enter some information into a form
Click Save
Behind the scenes, a new record is entered in the database
You want to edit the record you just entered, so you go to another web page. Behind the scenes it's passing the page a parameter with the Database ID of the row you just created
When you're running step 5 of the above test, the page parameter/Database ID is going to change each time.
The workflow/strategy I'm currently using is
Record a test using the above actions
Make a note of each place where a query string variable may change from run to run
Use a XPath or Regular Expression Extractor to pull the value out of a response and into a JMeter variable
Replace all appropriate instances of the hard-coded parameter with the above variable.
This works and can be automated to an extent. However, it can get tedious, is error prone, and fragile. Is there a better/commonly accepted way of handling this situation? (Or is this why most people just use JMeter to play back logs? (-;)
Sounds to me like your on the right track. The best that can be achieved by JMeter is to extract page variables with a regular expression or xpath post processor. However your absolutely correct in that this is not a scalable solution and becomes increasingly tricky to maintain or grow.
If you've reached is point then you may want to consider a tool which is more specialised for this sort of problem. Have a look web testing tool such as Watir, it will automatically handle changing post parameters; but you would still need to extract parameters if you need to do a database update but using Watir allows for better code reuse making the problem less painful.
We have had great success in testing similar scenarios with JMeter by storing parameters in JMeter Variables within a JDBC assertion. We then do our http get/post and use a BSF Assertion and javascript do complex validation of the response. Hope it helps