Which effect design should I use for my real world tiny Http/Spotify example?

[ParetoOptimalDev]

Working example of what I currently have here: https://github.com/ParetoOptimalDev/real-world-advanced-testing-using-effectful/tree/tinier

I'm trying to figure out how to properly design a simple program that queries spotify with effectful.

Before I get into details and give context my questions are:

• Is searchTrackProgram the correct approach over diverging "heavy" pure and IO interpreters
• Is it fine or even expected that the SearchTrack implementation in runSpotifyPure is so trivial?
• Is it expected that users will create many of their own effects or that they should likely default to using created ones?
• Is returning Right (Right [Track {trackName = "foo"}]) fine or is should I be perhaps catching HttpError and translating to SpotifyError?
• Would an example like this be good for effectful?

Note that these unanswered question have been the biggest reason keeping me from using effectful by default over mtl.

I think that this would have two effects:

data Http :: Effect where
  HttpGet  :: Url -> Http m Aeson.Value
makeEffect ''Http

data Spotify :: Effect where
  SearchTrack :: String -> Spotify m Aeson.Value

So now I could follow the example of ReadFile the filesystem example in the docs:

data FileSystem :: Effect where
  ReadFile  :: FilePath -> FileSystem m String

and create a pure interpreter:

runFileSystemPure
  :: Error FsError :> es
  => M.Map FilePath String
  -> Eff (FileSystem : es) a
  -> Eff es a
runFileSystemPure fs0 = reinterpret (evalState fs0) $ \_ -> \case
  ReadFile path -> gets (M.lookup path) >>= \case
    Just contents -> pure contents
    Nothing       -> throwError . FsError $ "File not found: "o ++ show path

This works pretty well for HttpGet where I use:

runHttpPure
  :: (Error HttpError :> es)
  => M.Map Url Aeson.Value
  -> Eff (Http : es) a
  -> Eff es a
runHttpPure db0 = reinterpret (evalState db0) $ \_ -> \case
  HttpGet url -> gets (M.lookup url) >>= \case
    Just response -> pure response
    Nothing       -> throwError . HttpError $ "no response found"

Then I use httpGet in the runSpotifyPure interpeter like:

runSpotifyPure
  :: (Http :> es, Error SpotifyError :> es)
  => Eff (Spotify : es) a
  -> Eff es a
runSpotifyPure = interpret $ \_ -> \case
  SearchTrack query -> do
    res <- httpGet (Url (playlistSearchUrl <> query))
    let tracks = res ^.. _Array % traversed % _String & fmap (Track . toString)
    if length tracks == 0 then
      throwError (SpotifyError "no tracks found")
      else pure tracks

The problem is that if I were to implement runSpotifyIO I would have two problems:

1. Duplication of this code:

    res <- httpGet (Url (playlistSearchUrl <> query))
    let tracks = res ^.. _Array % traversed % _String & fmap (Track . toString)
    if length tracks == 0 then
      throwError (SpotifyError "no tracks found")
      else pure tracks

2. the impedance mismatch that could occur between the pure and IO interpreters. This is important because frequently the biggest win for me is testing invariants with pure interpreters that mostly or 100% map to the IO implementation.

So this made me wonder if this means runSpotifyPure should instead be defined just as the quite trivial:

runSpotifyPure
  :: (Http :> es, Error SpotifyError :> es)
  => Eff (Spotify : es) a
  -> Eff es a
runSpotifyPure = interpret $ \_ -> \case
  SearchTrack query -> httpGet (Url (playlistSearchUrl <> query))

And then instead of my application using searchTrack created by makeEffect ''Spotify I could define and use:

searchTrackProgram :: (Http :> es, Error SpotifyError :> es, Spotify :> es) => String -> Eff es [Track]
searchTrackProgram query = do
  trackResponse <- searchTrack query
  let tracks = trackResponse ^.. _Array % traversed % _String & fmap (Track . toString)
  if length tracks == 0 then
    throwError (SpotifyError "no tracks found")
    else pure tracks

This way I could remove the duplication when using searchTrack arising from having two diverging interpreters and instead use searchTrackProgram for either case where I know it's exactly the same.

Overall I guess my goal is to be able to write a test like this:

main :: IO ()
main = hspec $ do
  describe "runSpotify" $ do
    it "searches for a track" $ do
      let db0 = fromList [("https://api.spotify.com/v1/search?type=track&q=", aesonQQ [aesonQQ|
            {
              "tracks": {
                "items": ["effectful"]
              }
            }
          |])]
          result = runPureEff $ runSpotify db0 $ searchTrack "effectful"
      result `shouldBe` aesonQQ [aesonQQ|
        {
          "items": ["effectful"]
        }
      |]

Edit: Potential answer to error question in #149 (comment)

[jhgarner]

The first example you give for runSpotifyPure looks like it should instead just be called runSpotify. If I understand it right, whether it's pure or not depends entirely on whether you're using the IO Http handler or the pure Http handler. That's also why you have so much code duplication between runSpotifyPure and the theoretical runSpotifyIO: they should be the same function!

If you want a unique runSpotifyPure, it probably shouldn't mention Http anywhere in the constraint. One benefit to having the Spotify effect instead of just having a searchTrackProgram function is that, when testing code which depends on the Spotify effect, you only have to mock out Spotify instead of mocking out Http too. This helps you focus your tests by ignoring implementation details like api endpoints and might be useful if you wanted to write a test which did other real Http effects while using mock data when communicating with Spotify.

[ParetoOptimalDev]

The first example you give for runSpotifyPure looks like it should instead just be called runSpotify. If I understand it right, whether it's pure or not depends entirely on whether you're using the IO Http handler or the pure Http handler. That's also why you have so much code duplication between runSpotifyPure and the theoretical runSpotifyIO: they should be the same function!

Yes... it should be a "program" rather than an interpreter and maybe that means I shouldn't even have the spotify effect...

If you want a unique runSpotifyPure, it probably shouldn't mention Http anywhere in the constraint. One benefit to having the Spotify effect instead of just having a searchTrackProgram function is that, when testing code which depends on the Spotify effect, you only have to mock out Spotify instead of mocking out Http too.

Wouldn't it internally need to use Http though? Maybe I'm not quite sure exactly how you mean this point.

[jhgarner]

Wouldn't it internally need to use Http though? Maybe I'm not quite sure exactly how you mean this point.

Why would a pure Spotify interpreter need to use Http? You could do:

runSpotifyPure
  :: Map String Aeson.Value
  -> Eff (Spotify : es) a
  -> Eff es a

And it would just return whatever Track it finds in the passed in Map. Now you can mock out the Spotify interactions directly instead of having to mock http calls. Another benefit is that whoever is writing the test won't need to know the format of the url that the Spotify implementation will use. You could have other tests just focused on verifying that your "real" runSpotify function makes the right http calls separate from the rest of your program's logic.

[the-dr-lazy]

Let's assume a program which should add all search results to the favorites:

searchTrackProgram :: (Http :> es, Error SpotifyError :> es, Spotify :> es) => String -> Eff es [Track]
addToFavorites :: (Http :> es, Error FavoriteError :> es, Favorite :> es) => Track -> Eff es ()

program :: ( Http :> es
           , Error SpotifyError :> es
           , Error FavoriteError :> es
           , Spotify :> es
           , Favorite :> es
           ) => String -> Eff es ()
program query = do
  tracks <- searchTrack query
  forM_ tracks addToFavorites

Here when it comes to testing, our test should be overwhelmed by handling effects i.e. Error and Http which by any means doesn't express our intention. So when I come around, something like searchTrackProgram and addToFavorites I think It is a total loss of the whole algebraic effect idea. It simply turned from an abstraction modeling system which helps us to understand and test the programs into just an effect tracking in the types.

Let's express our program in terms of resources, I think even the word spotify is leaky when we do so. We have some sort of track source which can be interpreted as spotify.

searchTrack :: TrackSourceManagement :> es => String -> Effect es [Track]
addToFavorites :: FavoritesManagement :> es => Track -> Eff es ()

program :: ( TrackSourceManagement :> es
           , FavoritesManagement :> es
           ) => String -> Eff es ()
program query = do
  tracks <- searchTrack query
  forM_ tracks addToFavorites

Testing this program is hilariously simple:

1. just generate a list of tracks.
2. run the TrackSourceManagement as it just return the generated tracks.
3. run the FavoritesManagement as writer
4. assert all tracks are logged into the writer

[ParetoOptimalDev]

Thanks.

I see what you mean I think.

However I'm not sure I agree spotify is leaky given that in my example the entire point of the application is interacting with spotify. I could see considering it leaky as reasonable if you wanted to support multiple services or had reason to believe you might move services.

With what I have in mind though, I think the ideal is to keep the concreteness of Spotify if that makes sense.