Haskell Weekly

>> Hello and welcome to the Haskell Weekly podcast. This is a show about Haskell, a purely functional programming. I'm your host Taylor Fausak. I'm the Director of Software Engineering at ACI Learning. With me today is Cameron Gera, one of the engineers on my team. Thanks for joining me today, Cam.

>> Thanks for having me, Taylor. I know it's been a long time since we've done a podcast. I know our faithful listeners are missing it, so I'm excited to be back today. You know, life gets busy sometimes in the summer for us, which seems to be the case. But today we're here to talk about something that's near and dear to us because it's something we've kind of created, more so Taylor than I. But something we also use on the regular. So about four weeks ago, Taylor had posted on Reddit about his blog post for this new library he created. So that is going to be our focus today. The library is called Witch, so if you're not familiar with it, we'll spend some time talking about that as well as just kind of talking about some of the motivations. How it solves some of our problems that we are facing on a day to day. So I'm really looking forward to this. So Taylor, you know, thanks for writing this library and writing this post and giving us an opportunity to talk about it.

>> For sure. It's my pleasure. As you mentioned, I finally got around to announcing this about four weeks ago, but we've actually been using this at ACI Learning for quite a bit longer than that. I want to say on the order of several months, but I don't remember exactly when I introduced it. But yeah, the intent of Witch was to make it easier to convert values between various types, which is something we end up doing a lot in our code base. And I feel most Haskell programmers probably do the same. And when I say convert between types, that means really basic conversions, like going from an Int16 into an Int, which is a totally safe conversion that you can do with something like fromIntegral. And then sometimes more complicated conversions, like going from an Int64 down into an Int16, which is something that can fail. And fromIntegral will type check for that, but it's actually unsafe. So this kind of a Swiss army knife function is the type of thing that Witch is meant to replace. And yeah, like I said, we've been using it in our code base. So Cam you've used it more kind of as an end-user of the library. What have been your thoughts on it so far?

>> Yeah, I think it's comes a lot of benefits, because you know, fromIntegral, like you said, like there's type checking that happens, but it's actually unsafe underneath. And it also doesn't necessarily depict and tell what kind of transformation you're making. You know, I think this, obviously you can always use type applications with this function to say, Hey, this is what I want to go to. But you know, this library Witch, it allows us to kind of explicitly say what we're trying to go into. And using type applications, it really, you know, makes it easier to read the code and kind of see how to do these, you know, transformations of, you know, in our test, in our data, in our code, all of this stuff that, you know, we're, we have JSON instances, we have, you know, database conversion functions and all those things require, you know, um, primitive types, excuse me. That, you know, you can generally transfer, you know, boil it down to. And so, you know, something like this allows you to convert from, you know, new type wrappers and between primitive types and all these things. And it also gives you another kind of, you know, way of maybe even trying some unsafe things and having a failure case, it'd be able to modify and handle and, you know, report on those kinds of edge cases that you may not see if you're using fromIntegral or, you know, toString or show or anything, you know, like all those things have their different caveats. And Witch kind of, you know, has been, has allowed me to not be so, kind of hesitant on what code's doing, especially when I'm reviewing code. Because, you know, nowadays I'm doing a lot more review than I am actually writing. And so it allows me to say, oh yeah, we're trying to go from, you know, a string into a course name or something along those lines. So aside for us that we use to depict meaning through the code base. You know, I think that stuff has been really helpful. And it's not as magical as like these catch all functions have been.

>> Yeah. Yeah, you said a lot there that I want to follow up on, but first I actually want to send another question to you, which is: So the name, Witch, w-i-t-c-h, like the female version of a wizard. Do you know why the library is called Witch?

>> Because it's magical.

>> That was certainly some of the inspiration, but actually in an earlier version of the library, the core kind of workhorse function that it provided was called cast. So I thought, you know what, what's something that casts between stuff. Oh, a witch can do that. So that's why I went with the name, but unfortunately these days, the cast function isn't in the library anymore. So the name just kind of seems like a non sequitor. But yeah.

>> We refined it, we iterated. Yeah, yeah. I forgot about, that we had it named cast, cause now I can think of this from and into, you know?

>> Yeah. So, yeah, that was the first thing that I wanted to follow up on was, so one of the, kind of, not really a problem, but a rough edges with a function like fromIntegral is that there are two type variables that you need to pin down when you use it, the source type and the target type. That's not what they're called in the function signature. I think they're just a and b, but you know, which type am I coming from? And which type am I going into? And usually from context, one or the other, and sometimes even both of those, are already pinned down for you. So you know that you're coming from an Int, but you're going into something polymorphic, or the other way around where you're starting with a polymorphic value and turning it into an Int. And that's why the Witch library exposes exactly those two functions, from and into. And they're exactly the same, except that their type variables are in opposite orders. And that's so that if you use type applications, you can usually get away with only mentioning one of the type variables. So you can say from @Int, which means I'm coming from an Int and I'm producing some polymorphic value, or you can say into @Int, which says I'm coming from some polymorphic value, but I'm ultimately gonna get an Int out of it. And that was just like for the ergonomics of our day to day usage, that was the easiest thing. And I'll say most of the time for us, we use into. Far more often than we use from.

>> Yeah. We use from more so in our tests because it's a little easier to be like, all right, we know that we're using overloaded strings here. So this is going to be text and we don't want it into, you know, some sort of test value and that way it's just done, we don't have to worry about it. But yeah, most of the time we want to end up using into, so we can say this is where we want to end up. And that way, you know, as it's going through that function, this is the value that's going to be produced.

>> Yep. And as I mentioned, a previous version of the library had a function called cast, which, um, it had all three functions at that time cast, from, and into. And the difference was that cast did not require a type application at all. But from and into did .And I've actually removed that restriction. So that's why cast went away, cause it's the same as from. But the reason I decided to move away from requiring type applications versus just strongly recommending them is that I felt like the library was trying to do too many things at once and I wanted to keep it really focused and say, no, it's just going do these conversions. And if you, as a consumer of the library, decide that you want to require a type applications, then that's on you. You know, if you can configure HLint to do that, good luck. If you want to write your own little wrapper library that requires them yourself, you can do that as well. So that was part of my motivation. And the other part was that requiring type applications is just kind of like a gross hack. There's nothing that you can decorate a function to easily say this needs a type application. There is another library. I forget the name of it right now, but there's another library that provides kind of a quick way to do that. So if you're using Witch and you want to require type applications, you can do that. But we just catch it and code review normally of, Hey, this, what type is this converting into? Can you throw a type application on there?

>> Yeah. And I wanted to touch back on some of those catch-all functions, like fromIntegral and things. You know, switching over to this library allowed us to get rid of a lot of little helper functions. Cause through code review, we'd see fromIntegral, we're like, ah, what's going on here? You know? And then we'd be like, can you make a separate function to kind of explicitly state, you know, going from an Int to an Int64 or, you know, anything like that. We would have just like a helper function that did that. And so we started to see that, you know, there's a lot of these helper functions popping up. And when you review your code, you're like, there's gotta be something here. Cause like it's a lot of similarities. There's a lot going on. So like what can we do to simplify that? Because all those little helper functions, they were just, you know, adding a little bit more to compile, which obviously it's a library, so we're still more or less compiling it, but it's, you know, much better, I feel like.

>> Yeah. And it's not so much the compilation time that was painful there. I'd say it was more just. Those functions aren't interesting to write. And when you see one, you know exactly what the implementation is or should be, but you still have to write it. And so we would have a function called like, you know, intToInt16 or integralToNatural or whatever the conversion was. And then we'd have the implementation, which for these primitive numeric types was often fromIntegral. So my goal with Witch was to have one function, which is from or into, they're the same thing, that we could use in all of those circumstances, but then have those type applications on there to say exactly what conversion we wanted. And then we can just delete all of those helper functions along the way. And we're not completely done with that transformation. It's still ongoing, but it's been very promising so far.

>> Yeah. And it's just been like for transforming types, from primitive to our new type wrappers, it's been super helpful as well. Uh, you know, because we don't have to worry so much about, well, what does this actually mean? What's going on here? Like, we don't have to have a toCourseName function or anything like that. Like it's just like into @CourseName and yeah, you're good to go.

>> Yeah, we've been talking a lot about primitive values, but one of the motivating factors for this was, we use new types all over the place in our code base to keep things distinct from, you know, when we pull them from the database to when we hand them off via our API. Or the other way around, when we receive it from the API, until we put it in the database, it's the same type all the way through. So as you said, course, name is a good example. We have many things in our system that can be named. And we want to make sure that we don't accidentally assign a course name to an episode, for example. So we have different new types for those things. And every time we made one of these new types in the past, we would have to write all of these conversion functions like stringToCourseName, textToCourseName, you know, all the various other, I mean, maybe those are the only ones there, but for integral types, there can be a lot. And when you write those functions, you have to decide, is this a total conversion or can it fail? So for instance, with course name, we want course names to have something in them. So they need to be non empty and they have to contain at least one non blank space character. Right? So that's another thing that Witch provides is a like partial conversion function. And we call it, tryFrom or tryInto. So that's something that says, the type you're coming from has values that can't be expressed into the type that you're going to. So you need to handle that somehow. And we can handle that in our code base with a regular exception, just throw like something in IO. Or an impure exception, if it's something that we feel won't ever happen, but you know, we have to appease the compiler. Or if it's a constant, we can lift it up to compile time and use a Template Haskell splice to drop it in there. So, yeah. Just to say, like, that was another motivation here is that some conversions work all the time and some conversions only work some of the time. And I really wanted to capture that distinction with two different type classes.

>> Yeah. And I think that really has allowed us to, not have to worry so much when we're doing a conversion, because we're like, oh yeah, let's try to go from Int64 to Int. And we're like, wait, there's not an instance for that in Witch. Like what's going on? It makes you think, oh, well this is an unsafe thing let's check in, tryFrom, right? Is that what the type class is? Right. So, you know, that gives us the ability to see what that, if that works. But if not, then we're not, you know, up a creek without a paddle.

>> Yeah, and especially as we're trying to onboard some new junior developers onto our code base, We want to push people toward using typed holes for development and say, if you don't know what goes here, just put an underscore there, and the compiler will tell you what things can fit in there. And unfortunately, using Int64 to Int, as an example, the compiler will happily suggest fromIntegral. And won't give you any indication that that is a potentially problematic, conversion. But now in that suggestion list two, or I think only one thing will pop up from Witch, and it will say unsafeTryFrom, which suggests to whoever's reading that list of, you know, typed hole suggestions that this conversion could fail. And so you may need to handle that somehow.

>> Right, it gives a little bit of guardrails around, you know, new engineers learning. So, or even, you know, they could be long-time engineers, but not familiar with Haskell. It could be another way to, you know, maybe, maybe introduce Haskell into your day to day work if you're not doing that day to day. Or you're, you know, trying to do some side project with a friend, you know, say, Hey, like let's, let's try Witch, and that way, you know, kind of, you can see some conversions that are safe and not going to create any issues. You know, I think that will provide a lot of stability for them. As they get onboarded and ramped up to speed. And, you know, I think there's a lot of good things in Witch like we've already kind of touched on. Another one thing I did want to touch on real quick before we just flew past it, was the fact that there's just like really great documentation on the Witch library. You know, there's explanations of what function it's really doing for the instance of, you know, Int to Int64. Or, you know, all these things, it gives you a clear definition of what's going on behind the scenes. So it's not as mystic. It's like, obviously you can always look at the source and see what it's doing, but, you know, you kind of gave a shortcut to just see what was really going on. So, you know, I think you could all also call this Shortcut, maybe IO, and take over Clubhouse's attempt.

>> Well thanks, I appreciate that. I definitely spent a lot of time on the documentation. Partly because I wanted it to be a resource for our own team to just point people to it, and say, you know, here are, here's some tips for how to convert stuff and why it works this way. But also because the actual implementation of Witch isn't that interesting. It's only a couple of lines of code for each type class. And most of the utility comes from either these related functions that I provide, and there's only a handful of them, or the documentation. Or just the existence of one instance or the other, like I was talking about earlier. If something has a From instance from a to b, then, you know, If it has a TryFrom instance from a to b, then, you know, that's not safe. And yeah, in the documentation for each instance between a pair of types, I link to the function that's used to implement that conversion. So for many of them, it ends up being fromIntegral. But for others, you know, like you can convert from a list into a set. And that uses Set.fromList. So this was my way of like, adding into that documentation, you know, maybe you don't actually want to use Witch in your code base, you just want to know, is there a way to convert between these things and you can go look at that list of instances and say, yeah, click on that. It's that function. Okay good, I'll go use that one instead. Or even if you are using Witch, but for a particular circumstance, you want to use a more concrete monomorphic conversion function rather than a type class method. Or just reading through and you're wondering, you know, how the heck is this thing implemented? You can see right there. But yeah, in general, if you haven't read the documentation, it's definitely meant to be approachable. And at the bottom I put a section that's much more conversational about kind of, why did I bother making this thing? What are some of the trade-offs it makes? What are some other things you might do instead? And what are some surprises? So like, one that has come up a couple of times is the Int32 into Int conversion is not total. And many people would probably expect it to be total. So go read the documentation, you can find out why it is the way it is.

>> Ooh, enticing them, I like it. Good, you know, we talk about these things on air and we just want to make sure that, you know, you always have access to see what we're talking about and validate for yourself. And obviously always ask questions. You know, we put this stuff out in the internet so that it can start conversations, not just, you know, be a echo chamber or anything along those lines. Like we want you guys to be able to express yourself. And as a Haskell community, we want to be there for one another. And you know, if we miss something great, if you have something you never thought about before that we said, great. You know, like all these things, you know, are here for you, as a listener of the Haskell Weekly podcast. And we just want to let you know that. But anyways, I wanted to get back to Witch, because I was kind of thinking how Witch actually can allow us to handle some conversions that allow us to not have to worry about deriving other instances. So for example, let's say, you know, I want to get a sum of episode lengths that would then equal the course length. You know, I could iterate over that list, get all of the integer or the Int values out of that, sum them up and then convert that to a course length rather than, you know, deriving an instance for Num on episode length.

>> Yeah, right. Yeah, and that's actually one. I feel like I keep saying, this was a motivation for making Witch. There were a lot of motivations, and one of them is that, some of the really core type classes like Num can be surprising to implement. So like for us, with episode length, as one of our integral new type wrappers, we actually don't want to implement Num for that because Num lets you multiply values together or divide them or add them, take the absolute value. And many of those operations don't make sense at all for episode lengths. If I multiply the length of this episode with that episode, what does that even mean? Episode length squared, like it's a nonsensical value. But we often want to do operations over these values, like you just mentioned. Sum all of them up. And, you know, we could implement that with a monoid and we may have done that with episode length, I don't remember. But yeah. With Witch you can say, convert all of them into an Int and then perform that operation on an Int and then turn that into something else. And we don't have to have a function in episode length that says convert it to an Int. And we don't have to have a function in course length that says convert it from an Int. We can handle that with the instances.

>> Yep, super nice. You know, we aren't sponsored by Which Which, just so you know. We have accidentally said that a few times. We joked about making a fake sponsorship for that, but yeah. We figured we don't want to get sued. So if you like Which Which, awesome. We're not talking about that though. So, sorry to let you down.

>> Yeah. It could be tricky if somebody has another library called, Which w-h-i-c-h, and then we'd have to talk about, well, which which are you talking about? But for now it's just the one. So Witch like a wizard.

>> Witch like a wizard, but female. Yeah. But yeah, I mean, I. I'm obviously a big fan of Witch. You know, we've been using it probably for four to five months now, I would say. You know, and like I said, we've had a ton of motivations for this. There's, you know, it's not gonna, not everybody's gonna agree. And that's okay. I know other, there's been other kind of conversion libraries that have come out that maybe have made some other surprising decisions. So, you know. We did what we felt was best for our day to day. If you want to use it, great. If you have feedback, that's also great. But we also know that we can't convince you to like our library. But well, Taylor's library. It's not mine, I didn't do it. So, you know, there's, you know, there's something that we're going to continue to maintain and, as other things come up, we could probably add some more. Is there anything else Taylor, that you're trying to add to Witch that you don't already have in there?

>> So I think I have most of the instances that I want in there. I'm sure there are more. But there are a couple problems with Witch that I haven't really figured out how to address. And the first is that, it's not completely clear when you should or should not provide an instance of something. And this is a problem because in general, with Haskell, your type class is supposed to have laws. And if there is an instance that meets all of the laws, then you should probably have that instance. And this can be a problematic viewpoint. So, the foldable type class, for instance. There is an implementation of that type class for tuples. So you can ask for the length of a tuple and it'll give you a surprising result. This is a huge flame war, so I don't really want to get into it. But, suffice to say that the type classes in Witch do not have laws. I've typed up a bunch of suggestions for them. But at the end of the day, it's really like a, it's just a call. Like, do I feel like this instance should belong or not? And that's guided me pretty well so far, but it leaves the door open of. So using tuples as an example, I could write an instance that says you can convert from a tuple of a and b into just a value of a. So you could, you could use from as sort of like, selector to get the first value out of a tuple. Yeah, exactly. You could have it to pull the second one out. Or imagine you have a big record with a bunch of fields on it that are all different types. So for us, like an episode that we'll have an episode ID and an episode length and an episode name. You could write a from instance that says, given an entire episode, I can give you one of those fields. So again, just doing a selector function. I haven't provided any of those yet, because my kind of guiding principle for which instances should be provided or not is, if you just saw a call to from @Episode in the code base, would you be able to tell based on context around there, what value is going to come out of that without really having to think about it too hard? And that's super wishy washy, I'll grant that. But it's worked well, have, you know, in any other programming language. Or not any other in, in like a, you know, dynamic programming language, like JavaScript, the automatic type conversions that the compiler will insert for you. That's kind of the type of thing I'm trying to target with Witch. Those types of conversions. So the fancier stuff that is technically, you know, makes sense and you could convince yourself, it's a good idea. I've shied away from that for now. You know, I'd be open to someone trying to change my mind there.

>> Yeah. So you're saying it's like, uh, you know, the wild, wild west with like an enforcer, right?

>> Yes.

>> Like no laws, but you know, there's a gut feeling that this is a good idea. This is not a good idea. So, okay.

>> And then the other kind of problem with Witch also revolves around type class instances. Unsurprisingly since that's really all that it provides. But it has to do with, let's say that I wanted to convert from a UUID into JSON. That is arguably something that the Witch library could provide. But in order to do that, the Witch library would have to depend on both the UUID library and let's say Aeson to do the JSON conversion. Usually when somebody makes a new library and there's kind of an ecosystem of libraries around it, you'll have something like, using lens as an example. You'll have lens or like lens-core, microlens or whatever, and then you have the lens library plus JSON. So you'll have lens-aeson. Or you'll have lens-uuid. Or whatever. And that works great because it's extending support from this one library to this other library. But with Witch it's problematic because you have to connect three libraries together rather than just two. So using the example from before, I would need a library that's Witch plus UUID plus Aeson to connect those together. And this obviously doesn't scale because as you add more libraries, kind of to the mix of things in ecosystem here, you have a huge number of libraries that you're generating. And this isn't a huge problem. And like, for us in our application that we're developing, we're not publishing a library after the fact. So we're kind of, the buck stops with us. And if we want to have one module in our application that just defines all of these random instances to connect stuff together, that's totally fine. But if I wanted to extract those into a library and publish them, that's where things get tricky because you want to have a minimal dependency footprint, but you also want to be maximally useful and provide instances for all these other libraries. And I have not found a good way to solve that problem. So if anyone has any ideas, please let me know. Yeah, I'm just at a loss there.

>> Yep. It's @TaylorFausak on Twitter or @HaskellWeekly on Twitter. You know, so check them out.

>> Yeah. And actually there was one final thing I wanted to go over Cam, which is something you touched on earlier, which is. If you're coming from another programming language, I think that Witch is especially useful, or if you're trying to convince someone else, Hey, Haskell is a good choice. I think that Witch could be a useful tool in your tool belt there. And I tried to make this really apparent in the documentation and the announcement that I pretty much stole this from the Rust programming language. They have exactly these, they're not type classes, they're traits. But they have From and TryFrom. And I thought it was a great idea. And I thought, why doesn't Haskell have that? Let's have that. But as I also mentioned, if you're coming from a dynamic language often, you know, somebody will want to convert an int into a string and that's kind of just automatic in a lot of languages. And with Witch it can sort of be automatic, you do have to put a function call there. But at least it just kind of does the right thing behind the scenes. You don't have to think about it too much. So if you're trying to convince someone to use Haskell, maybe throw a Witch into your argument there. I think it could be useful.

>> Yeah, for sure. I would definitely recommend as well. I've had a great time using it here and there and, it makes PR review a lot easier. There's not a lot of, mystery about what's happening there. So it just makes it a lot easier to work with, and I would definitely recommend it.

>> Yeah.

>> It's probably like the eight time I've said I recommend it, but that's okay.

>> And actually, I know I said that was the last thing, but that what you said reminded me. We use this a lot for new types and often there are multiple hops that we want to take between new types. So for instance, we have the let's say episode length as an example, we may want to go from, and Int64, that came from the database into an arbitrary precision, natural or integer, and then ultimately into the episode length type from there. Witch supports this with a utility function called via that just does two hops at once. So from and into will only go one step at a time, but via does two. And you only have to pick the type that's in the middle. So if you're in a situation where the source type and the target type are both already pinned down or clear from context, you can just say via some other type and it'll convert it for you. And we actually ended up doing this a lot with our various ID types that are in the database. Cause they're all just an Int at the end of the day or a UUID or whatever it is. And we have a custom type wrapper around that, that is for like doing the serialization to, and from our database. And then we have a type wrapper around that type wrapper for, you know, an episode ID or course ID. And so when we're pulling something from the database, we don't have to say into this, into that, we just say via, and it does the right thing. So that's super convenient.

>> Yeah, it is. It's definitely saved some time as well. Yeah. 10 out of 10 would recommend.

>> So I don't know if Cam recommends this library or not, but I sure do.

>> I don't know. It's only the ninth, 10th, 12th time.

>> Yeah. So yeah, I think those are all the things that I had to say about Witch. As is apparent from this talk, there were a lot of motivations, but it's been a really fun library to develop and really rewarding to kind of spread it through our code base and see all of the little paper cuts that it cleans up and makes nicer.

>> Yeah, ten out of ten. It's really nice for cleaning up all this old, like little tiny functions that are just really repetitive and there's a lot of boiler plate that really can just kind of get swept away. You know, and gives us that flexibility that we didn't really have before, because if you wanted to convert some value into a string and into an int or into a text, like that's two different functions for us, you know, now it's all right, we have two instances, but it's clear what's going on there.

>> All right. Well, have you got anything else to say about this Cam?

>> No, it just feels good to be back and recording some podcasts with you. I know we've had a little hiatus, so back from summer vacation now, you know, school starts soon and Haskell Weekly's raring back up.

>> Haskell Weekly, the only weekly podcast that comes out at least once a quarter.

>> That's a nice, nice tagline.

>> Yeah. All right. Well, thank you listeners for sticking with us and listening to the Haskell "Weekly" podcast. I have been your host Taylor Fausak and with me today was Cameron Gera. If you want to find out more about us, please check out our website, HaskellWeekly.News.

>> And we are brought to you by our employer, ITProTV, an ACI Learning company. They would like to offer you 30% off your subscription on ITPro.TV by using promo code HaskellWeekly30 at checkout. And I think that about does it for us. So if you are looking for a good library to try out, I would recommend Witch just for another time for good old sake. But, thanks for joining us and we'll see you next week.

>> Bye.