Hi I have simplified the problem to a method that simply copies the elements of one array to another array. My problem is that the final assert verifies yet the initial assert fails to verify even though I have a guard to ensure that initial assert only applies after the first time the loop is entered. Hence I think the final assert should imply the initial assert.
Any help much appreciated.
method simpImp(a:array<int>) returns (r:array<int>)
{
print "a ",a[..],"\n";
r := new int[a.Length];
var i:nat := 0;
while (i< a.Length)
decreases a.Length - i
invariant a.Length >= i
{
if (i> 0) { print " bool ",r[i-1] == a[i-1],"\n";}
else {print "*i ",i,"\n";}
//if (i> 0) { assert r[i-1] == a[i-1]; } //Fail
//assert (i>0) ==> r[i-1] == a[i-1]; //Fail
r[i] := a[i];
i:= i +1;
assert r[i-1] == a[i-1];
}
}
method Main() {
var a:array<int> := new int[5];
a[0],a[1],a[2],a[3],a[4] := 10,2,30,4,3;
var iny:= simpImp(a);
print "r ",iny[..],"\n";
}
The issue is that you need to add something to your loop invariant to "remember" any facts you need to carry between iterations. Dafny analyzes the loop body in isolation, assuming only the loop invariants at the beginning of the body. So unless you add the fact to the loop invariant, it will be "forgotten" by Dafny.
In your simplified example, adding
invariant i > 0 ==> r[i-1] == a[i-1]
to the loop causes the program to verify.
You can read more about loop invariants in the second half of this answer. Note that that discussion is phrased in terms of establishing that loop invariants are preserved, but the reasoning is mostly identical to trying to establish that an assertion inside the loop body is valid. Please feel free to ask further questions!
I have a dafny defined graph ADT (from this SO question) brought here again for completeness:
class Graph
{
var adjList : seq<seq<int>>;
constructor (adjListInput : seq<seq<int>>)
ensures adjList == adjListInput
{
adjList := adjListInput;
}
}
function ValidGraph(G : Graph) : bool
reads G
{
(forall u :: 0 <= u < |G.adjList| ==> forall v :: 0 <= v < |G.adjList[u]| ==> 0 <= G.adjList[u][v] < |G.adjList|) &&
(forall u :: 0 <= u < |G.adjList| ==> forall v,w :: 0 <= v < w < |G.adjList[u]| ==> G.adjList[u][v] != G.adjList[u][w])
}
method main()
{
var G : Graph := new Graph([[1,2],[0,2],[0,1]]);
var nonRelatedArray := new int[8];
var i := 0; while (i < 14)
{
// nonRelatedArray[3] := 55;
i := i + 1;
}
assert (ValidGraph(G));
}
If I remove the write comment to nonRelatedArray at index 3, I get an assertion violation, which is a bit weird because it seems reasonable that the memory model would be able to determine that nonRelatedArray is (well) non related to G.
You can fix this by adding modifies nonRelatedArray to the loop. The key to this modifies clause is that it does not mention G. So then Dafny knows that G will not be modified by the loop, so it will still be a valid graph.
It is a little confusing what happens if you leave off a modifies clause from a loop. If you don't do any writes to the heap (like when you comment out the write above), then Dafny (actually, Boogie) is able to automatically see that nothing is changed at all. But if you do any writes into the heap, Dafny's default modifies clause all of a sudden becomes "anything the surrounding scope is allowed to modify". If you want something other than these two defaults, you need to ask for it explicitly by giving a modifies clause.
I am having a problem with getting an invariant to be be maintained in dafny. The invariant is trying to maintain that a total is equal to the a recursive sum of elements in a sequence that match a given condition. What do i need to add/change to get this to verify. Here is my attempt:
datatype MovieTitle = A | B | C
class Movie {
var title: MovieTitle;
var run_time: int;
predicate Valid()
reads this
{
run_time >= 0
}
constructor(mt: MovieTitle, rt: int)
requires rt >= 0;
ensures Valid();
modifies this;
{
title := mt;
run_time := rt;
}
}
function movieSum(s: seq<Movie>, mt: MovieTitle, i: int): int
requires 0 <= i <= |s|;
decreases s, i;
reads s;
reads set x | x in s[..];
{
if |s| == 0 || i == 0 then 0
else if s[0].title == mt then s[0].run_time + movieSum(s[1..], mt, i - 1)
else movieSum(s[1..], mt, i - 1)
}
lemma {:induction s, mt, i} movieSumLemma(s: seq<Movie>, mt: MovieTitle, i: int)
requires 0 <= i < |s|;
ensures s[i].title == mt ==> (movieSum(s, mt, i) + s[i].run_time == movieSum(s, mt, i + 1)) &&
s[i].title != mt ==> (movieSum(s, mt, i) == movieSum(s, mt, i + 1));
decreases s;
{
}
method SumRuntimes(s: seq<Movie>)
{
var total := 0;
var i := 0;
while i < |s|
invariant 0 <= i <= |s|;
invariant total == movieSum(s, A, i);
decreases |s| - i;
{
if s[i].title == A {
total := total + s[i].run_time;
movieSumLemma(s, A, i);
}
i := i + 1;
}
}
Here invariant total == movieSum(s, A, i); won't hold. Any help in getting this to verify is appreciated. Thank you!
The problem is in function movieSum. You're both chopping off the initial element of s in the recursive call and decrementing i. That will not produce the sum of all the mt-movie elements.
You don't need the lemma. But its postcondition doesn't say what you may think it says. It currently has the form
ensures A ==> B && !A ==> C
The 2-character-wide && has higher binding power than the 3-character-wide ==>. So, your postcondition is saying
ensures A ==> ((B && !A) ==> C)
which simplifies to true. Instead, you want
ensures (A ==> B) && (!A ==> C)
which you can also write on two lines (for better readability):
ensures A ==> B
ensures !A ==> C
Your program also has a number of redundant decreases clauses and :induction attributes. I'm guessing you have added these from the blue information squiggles in the IDE. The blue squiggles are just FYI--Dafny is just trying to tell you what its default settings are.
It seems you may be using a very version of Dafny, or I expected you should have got a warning about the deprecated modifies this on the constructor.
I'm currently learning Dafny. I'm totally comfused by lemma and I dont know how to use it. The tutorial is not that helpful. What if I want to prove
count(a) <= |a|
how should I do it. Thanks for helping.
function count(a: seq<bool>): nat
ensures count(a) <= |a|;
{
if |a| == 0 then 0 else
(if a[0] then 1 else 0) + count(a[1..])
}
You've already proved it! You wrote the property you wanted as a postcondition of the function and Dafny verifies it without complaints. That's it.
You can also use a lemma to prove the property. Here's an example:
function count(a: seq<bool>): nat
{
if |a| == 0 then 0 else
(if a[0] then 1 else 0) + count(a[1..])
}
lemma CountProperty(a: seq<bool>)
ensures count(a) <= |a|
{
}
Again, Dafny verifies the lemma without issuing any complaints, so you have proved it!
It is not correct to assume that Dafny will always prove things for you automatically. Therefore, it's a good idea to learn how to write proofs manually, too. Here's a manual proof of this property. Just to make sure Dafny doesn't try to do induction automatically, I used a directive to turn it off (thus making our lives harder than Dafny normally would):
lemma {:induction false} CountProperty(a: seq<bool>)
ensures count(a) <= |a|
{
// Let's consider two cases, just like the definition of "count"
// considers two cases.
if |a| == 0 {
// In this case, we have:
assert count(a) == 0 && |a| == 0;
// so the postcondition follows easily.
} else {
// By the definition of "count", we have:
assert count(a) == (if a[0] then 1 else 0) + count(a[1..]);
// We know an upper bound on the first term of the addition:
assert (if a[0] then 1 else 0) <= 1;
// We can also obtain an upper bound on the second term by
// calling the lemma recursively. We do that here:
CountProperty(a[1..]);
// The call we just did gives us the following property:
assert count(a[1..]) <= |a[1..]|;
// Putting these two upper bounds together, we have:
assert count(a) <= 1 + |a[1..]|;
// We're almost done. We just need to relate |a[1..]| to |a|.
// This is easy:
assert |a[1..]| == |a| - 1;
// By the last two assertions, we now have:
assert count(a) <= 1 + |a| - 1;
// which is the postcondition we have to prove.
}
}
A nicer way to write a proof like this is to use a verified calculation, which Dafny calls "the calc statement":
lemma {:induction false} CountProperty(a: seq<bool>)
ensures count(a) <= |a|
{
if |a| == 0 {
// trivial
} else {
calc {
count(a);
== // def. count
(if a[0] then 1 else 0) + count(a[1..]);
<= // left term is bounded by 1
1 + count(a[1..]);
<= { CountProperty(a[1..]); } // induction hypothesis gives a bound for the right term
1 + |a[1..]|;
== { assert |a[1..]| == |a| - 1; }
|a|;
}
}
}
I hope that this gets you started.
Program safely,
Rustan
since few days I was struggling on how to proceed with PATCH request in Go REST API until I have found an article about using pointers and omitempty tag which I have populated and is working fine. Fine until I have realized I still have to build an UPDATE SQL query.
My struct looks like this:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
I am expecting a PATCH /resources/{resource-id} request containing such a request body:
{"description":"Some new description"}
In my handler I will build the Resource object this way (ignoring imports, ignoring error handling):
var resource Resource
resourceID, _ := mux.Vars(r)["resource-id"]
d := json.NewDecoder(r.Body)
d.Decode(&resource)
// at this point our resource object should only contain
// the Description field with the value from JSON in request body
Now, for normal UPDATE (PUT request) I would do this (simplified):
stmt, _ := db.Prepare(`UPDATE resources SET description = ?, name = ? WHERE resource_id = ?`)
res, _ := stmt.Exec(resource.Description, resource.Name, resourceID)
The problem with PATCH and omitempty tag is that the object might be missing multiple properties, thus I cannot just prepare a statement with hardcoded fields and placeholders... I will have to build it dynamically.
And here comes my question: how can I build such UPDATE query dynamically? In the best case I'd need some solution with identifying the set properties, getting their SQL field names (probably from the tags) and then I should be able to build the UPDATE query. I know I can use reflection to get the object properties but have no idea hot to get their sql tag name and of course I'd like to avoid using reflection here if possible... Or I could simply check for each property it is not nil, but in real life the structs are much bigger than provided example here...
Can somebody help me with this one? Did somebody already have to solve the same/similar situation?
SOLUTION:
Based on the answers here I was able to come up with this abstract solution. The SQLPatches method builds the SQLPatch struct from the given struct (so no concrete struct specific):
import (
"fmt"
"encoding/json"
"reflect"
"strings"
)
const tagname = "sql"
type SQLPatch struct {
Fields []string
Args []interface{}
}
func SQLPatches(resource interface{}) SQLPatch {
var sqlPatch SQLPatch
rType := reflect.TypeOf(resource)
rVal := reflect.ValueOf(resource)
n := rType.NumField()
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
for i := 0; i < n; i++ {
fType := rType.Field(i)
fVal := rVal.Field(i)
tag := fType.Tag.Get(tagname)
// skip nil properties (not going to be patched), skip unexported fields, skip fields to be skipped for SQL
if fVal.IsNil() || fType.PkgPath != "" || tag == "-" {
continue
}
// if no tag is set, use the field name
if tag == "" {
tag = fType.Name
}
// and make the tag lowercase in the end
tag = strings.ToLower(tag)
sqlPatch.Fields = append(sqlPatch.Fields, tag+" = ?")
var val reflect.Value
if fVal.Kind() == reflect.Ptr {
val = fVal.Elem()
} else {
val = fVal
}
switch val.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
sqlPatch.Args = append(sqlPatch.Args, val.Int())
case reflect.String:
sqlPatch.Args = append(sqlPatch.Args, val.String())
case reflect.Bool:
if val.Bool() {
sqlPatch.Args = append(sqlPatch.Args, 1)
} else {
sqlPatch.Args = append(sqlPatch.Args, 0)
}
}
}
return sqlPatch
}
Then I can simply call it like this:
type Resource struct {
Description *string `json:"description,omitempty"`
Name *string `json:"name,omitempty"`
}
func main() {
var r Resource
json.Unmarshal([]byte(`{"description": "new description"}`), &r)
sqlPatch := SQLPatches(r)
data, _ := json.Marshal(sqlPatch)
fmt.Printf("%s\n", data)
}
You can check it at Go Playground. The only problem here I see is that I allocate both the slices with the amount of fields in the passed struct, which may be 10, even though I might only want to patch one property in the end resulting in allocating more memory than needed... Any idea how to avoid this?
I recently had same problem. about PATCH and looking around found this article. It also makes references to the RFC 5789 where it says:
The difference between the PUT and PATCH requests is reflected in the way the server processes the enclosed entity to modify the resource identified by the Request-URI. In a PUT request, the enclosed entity is considered to be a modified version of the resource stored on the origin server, and the client is requesting that the stored version be replaced. With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version. The PATCH method affects the resource identified by the Request-URI, and it also MAY have side effects on other resources; i.e., new resources may be created, or existing ones modified, by the application of a PATCH.
e.g:
[
{ "op": "test", "path": "/a/b/c", "value": "foo" },
{ "op": "remove", "path": "/a/b/c" },
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
{ "op": "replace", "path": "/a/b/c", "value": 42 },
{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
{ "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]
This set of instructions should make it easier to build the update query.
EDIT
This is how you would obtain sql tags but you will have to use reflection:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
sp := "sort of string"
r := Resource{Description: &sp}
rt := reflect.TypeOf(r) // reflect.Type
rv := reflect.ValueOf(r) // reflect.Value
for i := 0; i < rv.NumField(); i++ { // Iterate over all the fields
if !rv.Field(i).IsNil() { // Check it is not nil
// Here you would do what you want to having the sql tag.
// Creating the query would be easy, however
// not sure you would execute the statement
fmt.Println(rt.Field(i).Tag.Get("sql")) // Output: description
}
}
I understand you don't want to use reflection, but still this may be a better answer than the previous one as you comment state.
EDIT 2:
About the allocation - read this guide lines of Effective Go about Data structures and Allocation:
// Here you are allocating an slice of 0 length with a capacity of n
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
With make(Type, Length, Capacity (optional))
Consider the following example:
// newly allocated zeroed value with Composite Literal
// length: 0
// capacity: 0
testSlice := []int{}
fmt.Println(len(testSlice), cap(testSlice)) // 0 0
fmt.Println(testSlice) // []
// newly allocated non zeroed value with make
// length: 0
// capacity: 10
testSlice = make([]int, 0, 10)
fmt.Println(len(testSlice), cap(testSlice)) // 0 10
fmt.Println(testSlice) // []
// newly allocated non zeroed value with make
// length: 2
// capacity: 4
testSlice = make([]int, 2, 4)
fmt.Println(len(testSlice), cap(testSlice)) // 2 4
fmt.Println(testSlice) // [0 0]
In your case, may want to the following:
// Replace this
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
// With this or simple omit the capacity in make above
sqlPatch.Fields = []string{}
sqlPatch.Args = []interface{}{}
// The allocation will go as follow: length - capacity
testSlice := []int{} // 0 - 0
testSlice = append(testSlice, 1) // 1 - 2
testSlice = append(testSlice, 1) // 2 - 2
testSlice = append(testSlice, 1) // 3 - 4
testSlice = append(testSlice, 1) // 4 - 4
testSlice = append(testSlice, 1) // 5 - 8
Alright, I think the solution I used back in 2016 was quite over-engineered for even more over-engineered problem and was completely unnecessary. The question asked here was very generalized, however we were building a solution that was able to build its SQL query on its own and based on the JSON object or query parameters and/or Headers sent in the request. And that to be as generic as possible.
Nowadays I think the best solution is to avoid PATCH unless truly necessary. And even then you still can use PUT and replace the whole resource with patched property/ies coming already from the client - i.e. not giving the client the option/possibility to send any PATCH request to your server and to deal with partial updates on their own.
However this is not always recommended, especially in cases of bigger objects to save some C02 by reducing the amount of redundant transmitted data. Whenever today if I need to enable a PATCH for the client I simply define what can be patched - this gives me clarity and the final struct.
Note that I am using a IETF documented JSON Merge Patch implementation. I consider that of JSON Patch (also documented by IETF) redundant as hypothetically we could replace the whole REST API by having one single JSON Patch endpoint and let clients control the resources via allowed operations. I also think the implementation of such JSON Patch on the server side is way more complicated. The only use-case I could think of using such implementation is if I was implementing a REST API over a file system...
So the struct may be defined as in my OP:
type ResourcePatch struct {
ResourceID some.UUID `json:"resource_id"`
Description *string `json:"description,omitempty"`
Name *string `json:"name,omitempty"`
}
In the handler func I'd decode the ID from the path into the ResourcePatch instance and unmarshall JSON from the request body into it, too.
Sending only this
{"description":"Some new description"}
to PATCH /resources/<UUID>
I should end up with with this object:
ResourcePatch
* ResourceID {"UUID"}
* Description {"Some new description"}
And now the magic: use a simple logic to build the query and exec parameters. For some it may seem tedious or repetitive or unclean for bigger PATCH objects, but my reply to this would be: if your PATCH object consists of more than 50% of the original resource' properties (or simply too many for your liking) use PUT and expect the clients to send (and replace) the whole resource instead.
It could look like this:
func (s Store) patchMyResource(r models.ResourcePatch) error {
q := `UPDATE resources SET `
qParts := make([]string, 0, 2)
args := make([]interface{}, 0, 2)
if r.Description != nil {
qParts = append(qParts, `description = ?`)
args = append(args, r.Description)
}
if r.Name != nil {
qParts = append(qParts, `name = ?`)
args = append(args, r.Name)
}
q += strings.Join(qParts, ',') + ` WHERE resource_id = ?`
args = append(args, r.ResourceID)
_, err := s.db.Exec(q, args...)
return err
}
I think there's nothing simpler and more effective. No reflection, no over-kills, reads quite good.
Struct tags are only visible through reflection, sorry.
If you don't want to use reflection (or, I think, even if you do), I think it is Go-like to define a function or method that "marshals" your struct into something that can easily be turned into a comma-separated list of SQL updates, and then use that. Build small things to help solve your problems.
For example given:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
You might define:
func (r Resource) SQLUpdates() SQLUpdates {
var s SQLUpdates
if (r.Name != nil) {
s.add("resource_id", *r.Name)
}
if (r.Description != nil) {
s.add("description", *r.Description)
}
}
where the type SQLUpdates looks something like this:
type SQLUpdates struct {
assignments []string
values []interface{}
}
func (s *SQLUpdates) add(key string, value interface{}) {
if (s.assignments == nil) {
s.assignments = make([]string, 0, 1)
}
if (s.values == nil) {
s.values = make([]interface{}, 0, 1)
}
s.assignments = append(s.assignments, fmt.Sprintf("%s = ?", key))
s.values = append(s.values, value)
}
func (s SQLUpdates) Assignments() string {
return strings.Join(s.assignments, ", ")
}
func (s SQLUpdates) Values() []interface{} {
return s.values
}
See it working (sorta) here: https://play.golang.org/p/IQAHgqfBRh
If you have deep structs-within-structs, it should be fairly easy to build on this. And if you change to an SQL engine that allows or encourages positional arguments like $1 instead of ?, it's easy to add that behavior to just the SQLUpdates struct without changing any code that used it.
For the purpose of getting arguments to pass to Exec, you would just expand the output of Values() with the ... operator.