Ignore invalid item when decoding list - elm

is it possible to ignore invalid items when decoding the list?
example: I have a Model
type Type
= A
| B
type alias Section =
{ sectionType : Type
, index : Int
}
getTypeFromString : String -> Maybe Type
getTypeFromString input =
case input of
“a” ->
Just A
“b” ->
Just B
_ ->
Nothing
decodeType : Decoder Type
decodeType =
Decode.string
|> Decode.andThen
(\str ->
case getTypeFromString str of
Just sectionType ->
Decode.succeed sectionType
Nothing ->
Decode.fail <| ("Unknown type" ++ str)
)
decodeSection : Decoder Section
decodeSection =
Decode.map2 Section
(Decode.field "type" decodeType)
(Decode.field "index" Decode.int)
if I decode the JSON
{
"sections": [{type: "A", index: 1}, {type: "invalid-type", index: 2}]
}
I expect my sections = [ {type = A, index= 1} ]

Generally the way you can deal with these is by decoding it to an Elm type that expresses the options, and then post processing with a map.
So for instance in your example, I would go for something like this:
decodeMaybeType : Decoder (Maybe Type)
decodeMaybeType =
Decode.string
|> Decode.map getTypeFromString
decodeMaybeSection : Decoder (Maybe Section)
decodeMaybeSection =
Decode.map2 (\maybeType index -> Maybe.map (\t -> Section t index) maybeType)
(Decode.field "type" decodeMaybeType)
(Decode.field "index" Decode.int)
decodeSections : Decoder (List Section)
decodeSections =
Decode.list decodeMaybeSection
|> Decode.map (List.filterMap identity)
NB: List.filterMap identity is a List (Maybe a) -> List a, it filters out the Nothing and gets rid of the Maybes in one go.

Given the comment by Quan Vo about one of many fields could be invalid, using Decode.oneOf might be a better fit.
You write the decoders for each field. If any field is illegal, the Section decoder fails and in oneOf, Nothing is returned.
(Here I am also using Json.Decode.Pipeline from NoRedInk).
import Json.Decode as Decode exposing (Decoder)
import Json.Decode.Pipeline exposing (required)
type Type
= A
| B
type alias Section =
{ sectionType : Type
, index : Int
}
getTypeFromString : String -> Maybe Type
getTypeFromString input =
case input |> String.toLower of
"a" ->
Just A
"b" ->
Just B
_ ->
Nothing
decodeType : Decoder Type
decodeType =
Decode.string
|> Decode.andThen
(\str ->
case getTypeFromString str of
Just sectionType ->
Decode.succeed sectionType
Nothing ->
Decode.fail <| ("Unknown type" ++ str)
)
decodeSection : Decoder Section
decodeSection =
Decode.succeed Section
|> required "type" decodeType
|> required "index" Decode.int
-- Either we succeed in decoding a Section or fail on some field.
decodeMaybeSection : Decoder (Maybe Section)
decodeMaybeSection =
Decode.oneOf
[ decodeSection |> Decode.map Just
, Decode.succeed Nothing
]
decodeSections : Decoder (List Section)
decodeSections =
Decode.list decodeMaybeSection
|> Decode.map (List.filterMap identity)

Related

Debugging a JSON decoder

I'm trying to test a decoder - but I only ever get back the default values. Apologies for the wall of text, but when I've tried smaller examples, they always work, so I'm guessing there's a stupid error in here somewhere.
I've been trying to figure out why this won't work for quite a while now, with no luck. The JSON appears to be valid (I have tried parsing it in JS and in online validators).
I've tried different methods of decoding the JSON, again with no luck.
Any help at all is very much appreciated. If anything else should be added to the question, please let me know (I'm new to elm, if you can't tell).
I'm trying to decode JSON which looks like this:
{
"fade": 1,
"colour": "Campbells Red",
"stock": 1,
"site": "",
"url": "",
"plastic":"DX",
"name":"aviar",
"seenAt":1612884837886,
"weight":175,
"compositeIdentifier":"aviar||Innova||DX||Campbells Red||175",
"manufacturer":"Innova",
"expiresAt":1615476837886,
"glide":3,
"turn":0,
"speed":2,
"price":8.99
}
My type looks like this:
type alias DiscSighting =
{ fade: Int
, colour: String
, stock: Int
, site: String
, url: String
, plastic: String
, name: String
, weight: Int
, compositeIdentifier: String
, manufacturer: String
, glide: Int
, turn: Int
, speed: Int
, price: Float
}
And my decoder looks like this:
discDecoder: Decoder DiscSighting
discDecoder =
succeed DiscSighting
|> andMap (field "fade" (int) |> (withDefault) -1)
|> andMap (field "colour" (string) |> (withDefault) "")
|> andMap (field "stock" (int) |> (withDefault) -1)
|> andMap (field "site" (string) |> (withDefault) "")
|> andMap (field "url" (string) |> (withDefault) "")
|> andMap (field "plastic" (string) |> (withDefault) "")
|> andMap (field "name" (string) |> (withDefault) "")
|> andMap (field "weight" (int) |> (withDefault) -1)
|> andMap (field "compositeIdentifier" (string) |> (withDefault) "")
|> andMap (field "manufacturer" (string) |> (withDefault) "")
|> andMap (field "glide" (int) |> (withDefault) -1)
|> andMap (field "turn" (int) |> (withDefault) -1)
|> andMap (field "speed" (int) |> (withDefault) -1)
|> andMap (field "price" (float) |> (withDefault) -1)
The error I get is due to a test failing (it returns the error side of the result and thus fails the test):
Err (Failure "Expecting an OBJECT with a field named price
)
In discDecoder I'm not sure what the definitions are for andMap and withDefault, but the optional function in the package NoRedInk/elm-json-decode-pipeline will work instead of andMap and withDefault:
discDecoder : Decoder DiscSighting
discDecoder =
succeed DiscSighting
|> optional "fade" int -1
|> optional "colour" string ""
|> optional "stock" int -1
|> optional "site" string ""
|> optional "url" string ""
|> optional "plastic" string ""
|> optional "name" string ""
|> optional "weight" int -1
|> optional "compositeIdentifier" string ""
|> optional "manufacturer" string ""
|> optional "glide" int -1
|> optional "turn" int -1
|> optional "speed" int -1
|> optional "price" float -1
Full working example here: https://ellie-app.com/ckYm8HhMJfKa1

Elm `update` is not executing the case statement when dropdown's value is changed

I am new to Elm. I am not able to call the update function once the dropdown value changes.
Scenario:
I have two dropdowns Grade and Environment. What I want is when Grade dropdown values change, the options of Environment dropdown will dependently change.
For example, if Grade dropdown value is 3 then the options of Environment should change to Imagine Math
gradeDropdown : String -> List String -> Html Msg
gradeDropdown grade grades =
let
buildOption =
gradeOption grade
in
select [ class "importWizard--gradeSelection", name "gradeSelection", onChange (UpdateStudent Grade) ]
(map buildOption grades)
gradeOption : String -> String -> Html Msg
gradeOption optSelectedVal temp =
let
optSelected =
temp == optSelectedVal
in
option [ value temp, selected optSelected ] [ text temp ]
environmentDropdown : Model -> String -> List String -> String -> Html Msg
environmentDropdown model learningEnvironment learningEnvironments selectedGrade =
let
buildOption =
environmentOption model learningEnvironment
blueprint_grades = ["PreKindergarten", "Kindergarten", "1"]
environmentDropdownOption =
if (selectedGrade == "" || (List.member selectedGrade blueprint_grades)) then
["Blueprint"]
else if (selectedGrade == "2") then
learningEnvironments
else
["Imagine Math"]
in
select [
class "importWizard--learningEnvironmentSelection"
, name "learningEnvironmentSelection"
, onChange (UpdateStudent LearningEnvironments)
]
(map buildOption environmentDropdownOption)
environmentOption : Model -> String -> String -> Html Msg
environmentOption model optSelectedVal temp =
let
optSelected =
temp == optSelectedVal
in
option [ value temp, selected optSelected ] [ text temp ]
And in Update
update : Flags -> Msg -> Model -> ( Model, Cmd Msg )
update flags message model =
case message of
UpdateStudent updateType newValue ->
let
validate =
validateStudent flags validatableFieldsForScreen
in
case updateType of
LastName ->
( validate { model | lastName = newValue } <| Just LastNameField, Cmd.none )
FirstName ->
( validate { model | firstName = newValue } <| Just FirstNameField, Cmd.none )
Sin ->
( validate { model | sin = newValue } <| Just SinField, Cmd.none )
Grade ->
( validate { model | grade = newValue, selectedGrade = newValue } Nothing, Cmd.none )
LearningEnvironments ->
( validate { model | learningEnvironments = newValue } Nothing, Cmd.none )
View:
, td [ class wizardTableInput ] [ gradeDropdown model.grade flags.grades ]
, td [ class wizardTableInput ] [ environmentDropdown model model.learningEnvironments flags.learningEnvironments model.selectedGrade ]
In this code, the environment dropdown's value is changing, however the model's value is not updated. From what I understand, I can see is environment dropdown's id re-rendered, but it is not updating the model's value of learningEnvironments. This means it is not executing the update function matching LearningEnvironments.
select widgets where the options change is one of the use cases for Html.Keyed.node
Use a helper function like the one bellow:
keyedSelect : (String -> a) -> String -> List ( String, String ) -> Html a
keyedSelect message selectedValue kvs =
let
toOption ( k, v ) =
( k
, option
[ value k
, selected (k == selectedValue)
, disabled (k == "-1")
]
[ text v ]
)
in
Keyed.node "select"
[ class "form-control", onChange message ]
(List.map toOption kvs)
I usually have a "Please select Foo" first option with the value -1 if the user never selected any of the options. This is why the code checks for -1 and disables the option. You can remove the disabled attribute if you don't need it.

How to batch multiple http calls together?

I have a report which I'm using as a basis to perform a number of Http calls to get details for each row.
LoadReport ->
( model
, Http.toTask (loadReport model.token model.page)
|> Task.andThen
(\report ->
Task.map (addProductDetailsResultsToReport report) (Task.sequence (prepareRequests model.token report))
)
|> Task.map filterOnlyMissingBarcodes
|> Task.attempt ProductData
)
The calls are sequenced and perform one after another which is very slow as I need to perform 20 calls in a row. I would like to do something analogues to JavaScript
Promise.all(prepareRequests)
I used to have them being processed using Cmd.Batch but then I couldn't find a way to know when the whole batch is finished loading, I need to load another batch if there are not enough rows on the screen.
I believe the solution already posted by Murph is correct. The following code is an example that demonstrates that solution by first getting a collection of photos from flickr and then getting captions for all those photos by batching a bunch of http get tasks. Two lists are maintained in the model - untitled photos and titled photos. As the responses to the http gets come in, the appropriate photo is added to titled photos with the title assigned.
In this example the code can tell that all the gets have been responded to when the length of the titled list is the same as the length of the untitled list but it could just as easily have been done by removing photos from the untitled list until it is empty.
Here's a working demo
module Main exposing (..)
import Browser
import Html exposing (Html, div, text)
import Html.Attributes as HA
import Http
import Json.Decode as DC
import Svg
import Svg.Attributes as SA
import Task
type Msg
= SetPhotos (Result Http.Error (List Photo))
| SetDescription (Result Http.Error ( String, String ))
main : Program () Model Msg
main =
Browser.element
{ init = init
, view = view
, update = update
, subscriptions = \m -> Sub.none
}
type alias Model =
Result Http.Error
{ untitled : List Photo
, titled : List Photo
}
decodeUser : DC.Decoder String
decodeUser =
DC.at [ "user", "id" ] DC.string
type alias Photo =
{ id : String
, secret : String
, server : String
, farm : Int
, description : Maybe String
}
-- Create a Photo record from info retrieved from flickr api.
-- Get description later
initPhoto : String -> String -> String -> Int -> Photo
initPhoto id sec ser farm =
Photo id sec ser farm Nothing
decodePhotoList : DC.Decoder (List Photo)
decodePhotoList =
DC.list <|
DC.map4 initPhoto
(DC.at [ "id" ] DC.string)
(DC.at [ "secret" ] DC.string)
(DC.at [ "server" ] DC.string)
(DC.at [ "farm" ] DC.int)
-- Decode photos from "flickr.people.getPublicPhotos" request.
decodePhotos : DC.Decoder (List Photo)
decodePhotos =
DC.at [ "photos", "photo" ] decodePhotoList
-- Decode descripion of photo from "flickr.photos.getInfo" request.
decodePhotoDescription : DC.Decoder String
decodePhotoDescription =
DC.at [ "photo", "description", "_content" ] DC.string
-- api key from flickr. Anyone who clones this project should
-- get their own api key.
apiKey : String
apiKey =
"e9d3fdd5c2e26f9ebd13f4983cf727db"
flickrRestServices : String
flickrRestServices =
"https://api.flickr.com/services/rest/?"
noJsonCallback : String
noJsonCallback =
"&format=json&nojsoncallback=1"
userUrl : String -> String
userUrl name =
flickrRestServices
++ "&method=flickr.people.findByUserName"
++ "&api_key="
++ apiKey
++ "&username="
++ name
++ noJsonCallback
publicPhotosUrl : String -> String
publicPhotosUrl uid =
flickrRestServices
++ "&method=flickr.people.getPublicPhotos"
++ "&api_key="
++ apiKey
++ "&user_id="
++ uid
++ noJsonCallback
photoInfoUrl : String -> String
photoInfoUrl photo =
flickrRestServices
++ "&method=flickr.photos.getInfo"
++ "&api_key="
++ apiKey
++ "&photo_id="
++ photo
++ noJsonCallback
-- Cmd to get photo description from flickr.
-- Package results as SetDescription message.
-- Save the photo id with Task.map to apply the description to the right photo
setDescriptionCmd : Photo -> Cmd Msg
setDescriptionCmd dp =
case dp.description of
Nothing ->
Task.attempt SetDescription (Task.map (\s -> ( dp.id, s )) <| Http.toTask <| Http.get (photoInfoUrl dp.id) decodePhotoDescription)
Just des ->
Cmd.none
-- Cmd to get users public photos from flickr.
-- Package results as SetPhotos message.
getPhotosCmd : String -> Cmd Msg
getPhotosCmd name =
let
req =
Http.get (userUrl name) decodeUser
userTask =
Http.toTask req
publicPhotosTask uid =
Http.toTask (Http.get (publicPhotosUrl uid) decodePhotos)
userPhotosTask =
userTask |> Task.andThen publicPhotosTask
in
Task.attempt SetPhotos userPhotosTask
init : () -> ( Model, Cmd Msg )
init _ =
( Ok
{ untitled = []
, titled = []
}
, getPhotosCmd "elmDemo" -- flickr user name
)
-- UPDATE
update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
case msg of
SetPhotos (Ok photos) ->
( Ok
{ untitled = photos
, titled = []
}
, Cmd.batch <| List.map setDescriptionCmd photos
)
SetPhotos (Err e) ->
( Err e
, Cmd.none
)
-- Update description of the photo with matching id.
SetDescription (Ok ( photoId, desc )) ->
case model of
Ok photos ->
let
justTitled =
photos.untitled
|> List.filter (\ph -> ph.id == photoId)
|> List.map (\ph -> { ph | description = Just desc })
newTitled = photos.titled ++ justTitled
newPhotos = { photos | titled = newTitled }
in
( Ok newPhotos
, if
List.length newPhotos.titled
== List.length newPhotos.untitled
then
Cmd.none -- Could do something else here.
else
Cmd.none
)
Err e ->
( Err e
, Cmd.none
)
SetDescription (Err e) ->
( Err e
, Cmd.none
)
-- Compute a photo URL from a Photo record.
-- per: https://www.flickr.com/services/api/misc.urls.html
photoUrl : Photo -> String
photoUrl ps =
"https://farm"
++ String.fromInt ps.farm
++ ".staticflickr.com/"
++ ps.server
++ "/"
++ ps.id
++ "_"
++ ps.secret
++ "_b.jpg"
-- show an image and description if available.
viewPhoto : Photo -> Html Msg
viewPhoto ps =
div
[ HA.style "height" "20%"
, HA.style "width" "20%"
, HA.style "margin" "0"
]
[ div
[ HA.style "height" "90%"
, HA.style "width" "100%"
, HA.style "margin" "0"
]
[ Svg.svg
[ SA.version "1.1"
, SA.width "100%"
, SA.height "100%"
, SA.viewBox "-1 -0.6 2 1.2"
, SA.preserveAspectRatio "none"
]
[ Svg.image
[ SA.xlinkHref (photoUrl ps)
, SA.x "-1"
, SA.y "-0.6"
, SA.width "2"
, SA.height "1.2"
]
[]
]
]
, div
[ HA.style "height" "10%"
, HA.style "width" "100%"
, HA.style "margin" "0"
]
[ div
[ HA.style "text-align" "center" ]
[ text <| Maybe.withDefault "" ps.description ]
]
]
-- Draw an image or display the reason the image is not available.
view : Model -> Html Msg
view model =
case model of
Err s ->
text "Error: "
Ok photos ->
div []
[ div [] [ text "UNTITLED" ]
, div [] (List.map viewPhoto photos.untitled)
, div [] [ text "TITLED" ]
, div [] (List.map viewPhoto photos.titled)
]
Random thought:
Given that you will get a response back for each call you can keep track of the calls received by creating a collection of expected responses before calling batch and then removing the appropriate item from the collection each time a response is received.
At the point at which that collection is empty you've received all the responses and can fire off the next batch.
There are any number of variations on this pattern that should achieve the desired result. (And probably other patterns that would work just as well.)

How do I write a decoder to map a list of values for a custom data type?

I'm struggling to write a decoder for a list of links:
listOfLinksDecoder : Decoder (List JsonLink)
listOfLinksDecoder =
Decode.map (List JsonLink)
(field "Links" <| Decode.list linkDecoder)
Error:
Decode.map (List JsonLink)
Cannot find variable List
Please note that I have been successful in writing a decoder for a single link:
linkDecoder : Decoder JsonLink
linkDecoder =
Decode.map6 JsonLink
(field "Profile" profileDecoder)
(field "Title" Decode.string)
(field "Url" Decode.string)
(field "ContentType" Decode.string)
(field "Topics" <| Decode.list topicDecoder)
(field "IsFeatured" Decode.bool)
Please note that I attempted to search this documentation. However, I was still unable to find an example for my case.
Appendix:
topicLinks : Id -> Topic -> ContentType -> (Result Http.Error (List JsonLink) -> msg) -> Cmd msg
topicLinks providerId topic contentType msg =
let
url =
baseUrl ++ (getId providerId) ++ "/" ++ "topiclinks"
body =
encodeId providerId |> Http.jsonBody
request =
Http.post url body linksDecoder
in
Http.send msg request
You don't need to map, you can just do:
listOfLinksDecoder : Decoder (List JsonLink)
listOfLinksDecoder =
field "Links" <| Decode.list linkDecoder
Also note that Cannot find variable List error is because List is a type and not a constructor/function.

How do I implement a decoder that calls itself due to one of its fields having the same type?

How do I implement a decoder that calls itself due to one of its fields having the same type?
providerDecoder : Decoder JsonProvider
providerDecoder =
Decode.map6 JsonProvider
(field "Profile" profileDecoder)
(field "Topics" <| Decode.list topicDecoder)
(field "Links" <| linksDecoder)
(field "RecentLinks" <| Decode.list linkDecoder)
(field "Subscriptions" <| Decode.list providerDecoder)
(field "Followers" <| Decode.list providerDecoder)
The following lines are causing issues:
(field "Subscriptions" <| Decode.list providerDecoder)
(field "Followers" <| Decode.list providerDecoder)
providerDecoder is defined directly in terms of itself, causing an
infinite
In conclusion, I am not sure how to resolve this error while still preserving the JsonProvider type.
Appendix:
type JsonProvider
= JsonProvider
{ profile : JsonProfile
, topics : List JsonTopic
, links : JsonLinks
, recentLinks : List JsonLink
, subscriptions : List JsonProvider
, followers : List JsonProvider
}
When you write recursive JSON decoders, you usually have to rely on Json.Decode.lazy. You can write those two lines as this:
(field "Subscriptions" <| Decode.list (Decode.lazy (\_ -> providerDecoder)))
(field "Followers" <| Decode.list (Decode.lazy (\_ -> providerDecoder)))
Once you change that you'll see another error message pop up about the types not matching up, and that's because you're using a single constructor union type that has a record as an argument (which is necessary when writing recursive record types). In this case I usually separate out the constructor and record type like this:
type JsonProvider
= JsonProvider JsonProviderFields
type alias JsonProviderFields =
{ profile : JsonProfile
, topics : List JsonTopic
, links : JsonLinks
, recentLinks : List JsonLink
, subscriptions : List JsonProvider
, followers : List JsonProvider
}
Now you can rewrite the provider decoder to first decode the JsonProviderFields record, then map it to a JsonProvider:
providerDecoder : Decoder JsonProvider
providerDecoder =
Decode.map6 JsonProviderFields
(field "Profile" profileDecoder)
(field "Topics" <| Decode.list topicDecoder)
(field "Links" <| linksDecoder)
(field "RecentLinks" <| Decode.list linkDecoder)
(field "Subscriptions" <| Decode.list (Decode.lazy (\_ -> providerDecoder)))
(field "Followers" <| Decode.list (Decode.lazy (\_ -> providerDecoder)))
|> Decode.map JsonProvider