asoft
sea

I think we can manage

tl’dr: sbt Managed dependencies using git submodules on heroku. It can be done.

A post about source compatibility recently stirred up some commotion in the Scala community.

I too have felt some of the compatibility pain, but even though I have only been dancing the Odersky waltz for only a fraction of the time it takes to get on scala’s Dancing with the Stars, I believe I have seen hope. And I believe that source is the answer.

The source of the problem

In recent times, the common tech wisdom says that after 6-7 years, you are pretty much old news. The unfortunate news I have for you is that I think this is actually the time it takes to mature into something both dependable and beautiful. Younger, possibly cuter, programming languages will eventually learn that they will need to evolve and mature too, assuming they stick around for that long.

Evolution is in Scala’s blood. Some languages resist evolution in order to honor dead ancestors by carrying around spirits of the past. I don’t see Scala as a language that attends that same summer camp. As such, during its youth, Scala traded dead weight for a better design. The compromise was backwards binary compatibility. This obviously will cause issues with dependencies, hence the great debacle.

Simple, the way nature intended it

It wasn’t long before Sbt hit the Scala scene lookin’ all hipster-like from the frames of xml-configured eye glasses and roaming ants. Sbt gave us a number of things, one of which was the ability to build Scala software using Scala source code. The issues with source compatibility were known at that time too. Sbt’s the author couldn’t have made is simpler for library authors to wrangle the the issue of publishing binaries for their libraries across any combination of Scala versions you can think of.

The hat trick was to take something like

 publish

and turn into something like

+ publish

To this day, it baffles me that some library authors haven’t caught on to this and even complain about it. I can’t see it getting any more simple that adding a single operator. Although I can see some talking issue with it being an operator. ( Just kidding guys :) )

But there was something even more beautiful that came out of those early versions of Sbt: source dependencies, dependencies that could be managed for you and be compiled right along side your code using the same version of Scala.

Fast forward, Let’s git to the point

The latest versions of Sbt also support source dependencies, but through a much more powerful tool, git.

In addition to resolving dependencies through ivy/maven identifiers, Sbt now provides a simple way for declaring managed source dependencies through git uris, another under-rated, and probably under used featured among library authors.

 lazy val mahSourceDep = uri("git://github.com/the/awesome.git")

Here’s the nice thing about git. It’s all about the versions!

lazy val mahSourceDep = uri("git://github.com/the/awesome.git#mahbranch")

lazy val mahSourceDep2 = uri("git://github.com/the/awesome2.git#mahtag")

lazy val mahSourceDep3 = uri("git://github.com/the/awesome3.git#mahsha")

While I agree mvn published-versions of libraries work well for binary dependencies, I can’t really see authors publishing a binary for every bug fix/commit. But you know where I can see them publishing them to often? git.

As such, for agile development and a community-driven workflow, the more I look to a system mostly defined in terms of xml and reversed domain names that don’t really mean anything anymore, the more it feels a bit, antiquated? Don’t quote me on that! I know you’ve been doing this since before I was in short pants. I’m just sayin’.

Heroku me loco

That’s not the only way to play the source fiddle. Once you have your highly-scalable, fault- tolerant web 4.8 online pet store ready to go, the next step is to get it hosted.

My recent love affair with hosting has been with Heroku. Why is Heroku different than other hosting providers? Heroku gets developers. Heroku gets developer workflows. Heroku gets git.

How do you deploy am app on Heroku?

git push heroku master

KA-POW!

Don’t worry Scala coders, Heroku speaks many dialects of sbt.version.

So now that I’ve got you all PUMPED about source dependencies and Heroku. I’m going to tell you a stort fail story. And then I’m going to tell you another story. About not failing. About source. And git. And not failing.

I recently refitted a GAE armored app with the scantily garb of nothingness Heroku requires and tripped on a bump in the cloud.

The way Sbt git source dependencies work is that it clones them into a staging area, outside of your project, before compilation. So what’s the problem with that? When you push to Heroku you are actually triggering the compilation of something called a slug. Heroku is smart enough to know how to detect your flavor of slug (language), but there is one thing you should be aware of.

There ain’t no writing to disk in a cloud an their ain’t no special treatment for staging a clone out side of your slugs play pen!

insert crazy angry face here

Well that’s just great. All this show and dance and no payout. Well, that’s not the end of the story my friends.

If there is once thing for certain with Sbt’s new awesome design, it’s that you’ve got options™.

A good friend of mine pointed out that you can actually use git as unintended with Sbt for profit.

Here are the secret ingredients.

1) Git will, by default, allow you to create submodules in any subdirectory of a repo.

2) Sbt will, by default, resolve “unmanaged dependencies” under your projects ‘lib’ directory.

3) Heroku will allow you enable submodules

So if you declare a submodule for your awesome dependency

 $ git submodule add awesome git://github.com/the/awesome.git#mahbranch lib/awesome
 $ git submodule init
 $ git submodule update

And tell sbt to depend on it

  lazy val mahAwesomeDep = file("lib/awesome")

and enable gitsubmodules on Heroku

  $ heroku labs:enable git_submodules

You may just get your wings after all

  git push heroku master

Don’t believe me? I’m doing it now.

sticking ? where it doesn’t belong

One fondness I have in common with Ruby, Clojure, and even Coffeescript is a syntactic idiom of ending the names of functions and methods that return a false value with a ?.

The idea is to immediately convey the intent of the function by means of an expectation you’ve developed reading natural language. When you read something that ends in a ?, you intuitively assume, without thinking, that sentence will result in a yes or no answer. Agree?

In Java land, the idiom is to prepend methods that return a boolean value with is. An example would be isJavaLike(). The idiomatic way in the languages listed above for expressing this would be javaLike?. I prefer predicate methods that end with ? because they read nicer and follow consistent rules. Let’s say you have a method is() in Java that returns a boolean value. If you wanted to be consistent you’d say isIs(). Chances are, you would break consistency and just stick with is() because isIs() just sounds silly. In the languages mentioned above, is? is idiomatically consistent and still readable.

flex time

In my spare time I like to whittle projects made of Scala. Scala is a super-flexible language that allows you to create fluent DSL’s by combining first class functions with a relatively open list of characters that you are free to use for method names.

The only complaint I have is that, while the method definition

def __^|(i: Int) = i % 2 == 0

is syntactically valid

def even?(i: Int) = i % 2 == 0

is not!

If you paste that in a Scala REPL you are likely to get this error

<console>:1: error: '=' expected but identifier found.
       def even?(i: Int) = i % 2 == 0
               ^

Method names in Scala can not end with ?.

How displeasing. :/

Some Scala libraries like Lift have an awkward response to this grammar constraint. In Lift, it is idiomatic to end method names that return boolean values with _?. The even method above would be defined as even_? in Lift. This feels very awkward to me because there is a disconnect in consistency with the mixing of camelCase method names with snake_casing. Something about fooBarBaz_? just doesn’t feel right to me. Does it feelRight_toYou_?.

I should mention at this point that if you read the Variable Declarations and Definitions section of the language spec you will see the use of def x_= (y: T ): Unit, suggesting the using of _ as suffix for mutating members. This has always felt awkward to me as well. In these cases, I prefer returning a new copy of type Foo given the method def x(y: T): Foo which retains the method name, unchanged, verses appending a _ and mutating the instance.

But I digress…

Well, Scala is a flexible language so of course there are ways to work around this.

And I’m not talking about something like

 def `even?`(n: Int) = n % 2 == 0

In languages like Ruby, you can monkey patch a class. In Scala, you use types to implicitly decorate values.

My first thought was to do something like

implicit def questionable[T](b: Boolean) = new { def ? = b }

This will enable you decorate any expression that returns a boolean value with a ?.

So, you could say something like

if(true?) ...

But that flavoring doesn’t necessarily have the same sweetness to it with methods that take arguments.

if(even(3)?) ...

The ? must come after then expression.

Well then, how about

 implicit def questionablefun[T](b: T => Boolean) = new { def ? = b }

 def pred[T](f: T => Boolean)(t: T) = f(t)

Let’s hear if for first class functions and currying!

Given the tools above, we can now assign a curried function that returns a boolean to a value and end that with a ? prior to applying the second argument.

val even = pred((i: Int) => i % 2 == 0)_

if(even?(2)) ...

Ha. Take that silly syntax grammar rules!

Now, of course I would not recommend doing something like all that in a production environment, but it does put a smile on my face just knowing that I can. For the time being I will be content with neither is, ?, nor _? and just stick unadorned intention revealing method names with context

if(allElseFails(Seq(4, 8, 15, 16, 23, 42))) ...

Works for me.

magic match sticks and burnt fingers

First class pattern matching support is one of the many ways Scala stole my heart. I now use pattern matching as a superior refactoring and design tool in place of most every use case where I would historically insert another stray branch into a nest of ifs or query methods. Scala’s match keyword does what most languages switch or case keywords should do. Some may argue there’s too much magic behind it but once you understand that happens behind the curtains with unapply, most of that unfounded distrust goes away. For me, it’s just just the right amount of magic for me enjoy sticking my hand in the magician’s hat without the fear of uncertainty.

That doesn’t ways mean what my hand draws from the magicians hat is always what I expect.

Let’s look an extremely contrived example of using pattern matching.

def isRabbit(contents: String) =
  contents match {
    case "Buggs" => true
    case "Roger" => true
    case "Jessica" => false
    case _ => false 
 }

I’ve seen this trick enough to know that when I put my hand in, I have a pretty good idea of what I’ll get out depending on it’s contents. When see the magician stuff Buggs bunny into his hat before the show, I know I will pull out a rabbit. When I see the magician stuff Jessica Rabbit into his hat before the show, I know I will not being pulling out a rabbit. We’ve all been to that show before.

Now. We are good at extracting the hardcoded string values of our applications are’t we? Let’s extract thems that be causin all the code smell.

val buggs = "Buggs"
val roger = "Roger"
val jess = "Jessica"

def isRabbit(contents: String) =
  contents match {
    case buggs => true
    case roger => true
    case jess => false
    case _ => false 
 }

Can you guess what the contents of the hat will reveal when the magician puts “Roger” into the hat before the show? You sure? Okay. Now paste the code above into your nearest scala repl.

Ouch! Looks like you just got burnt. What’s with all this error: unreachable code business? Scala is supposed to prevent me from having buggs in my program, isn’t it?

Unbeknownst to me until recently is a little bit of knowledge that I hope to bestow upon the likes of you. So quiet the television and come sit close to the fire. Scala’s pattern matching engine will not treat lowercase variable names the same as will their value in match clauses. Don’t assume you can just use variables in the values’ stead without getting burnt. But, there is hope for sorry souls.

Scala has a convention for naming contants and so does English for its proper nouns: the first letter should be upper-cased.

Let’s revisit this ridiculous example so I can get to the point. Let’s rename those proper nouns using upper case identifiers.

val Buggs = "Buggs"
val Roger = "Roger"
val Jess = "Jessica"

def isRabbit(contents: String) =
  contents match {
    case Buggs => true
    case Roger => true
    case Jess => false
    case _ => false 
 }

Ah, ha! It totally works now. Why? Because scala treats pattern match case arguments that are identifiers as extractors and by convention an identifier starting with an uppercase letter is assumed to be a constant identifier. A constant can statically be compiled in bytecode because it will not change. A lowercase identifier by convention is not, and therefore is a lesser citizen and can be assumed to vary at runtime and cannot be statically compiled as a pattern match clause. So why do inline, hard coded strings work then? Duh. Strings are immutable and can not change so they can be statically compiled with the pattern match clause. With this information in hand, we should hopefully see why the semi-compiled version using lower case variable names errors out. The remaining case statements would never be reached and Scala thankfully lets us know this at compile time. One of the benefits you are afforded in a statically compiled language. At least this is my hypothesis :)

Before this magic show ends, I’d like to point out a lesser know feature of Scala’s syntax. Enter: the backtick! In some rare cases where you can just not stand to not use lower case variable names or when you get the itch to name something using a language keyword, you can wrap the identifier in a set of backticks. This will say to Scala: “Hey, I know this is wrong but give me a break. The magic business hasn’t been all that great lately”. Scala, being the nice language it is will reply: “Okay, I’ll let is one go for now”.

So if you absolutely wanted to use lower case identifiers in pattern match clauses, you can sneakily get away with the following.

def isRabbit(contents: String) =
  contents match {
    case `buggs` => true
    case `roger` => true
    case `jess` => false
    case _ => false 
 }

An surprisingly you can also get away with tomfoolery like this.

val `val` = "ue"
println(`val`)

the magician bows

I’d like to thank @boia01 and @jorgeortiz85 for some helpful insight on some frustration I had with an almost unrelated issue but with a totally related answer.

Enjoy the show!