Pedro Matiello

Announcement: me.pmatiello/openai-api

2023-04-25T22:30:00+00:00

me.pmatiello/openai-api is a pure-Clojure wrapper around the OpenAI API, offering various functions for interacting with the API’s capabilities. These include text generation, image generation and editing, embeddings, audio transcription and translation, file management, fine-tuning, and content moderation.

(require '[me.pmatiello.openai-api.api :as openai])

(def credentials
  (openai/credentials api-key))

(openai/chat {:model    "gpt-3.5-turbo"
              :messages [{:role    "user"
                          :content "Fix: (println \"hello"}]}
             credentials)

Notice: This is not an official OpenAI project nor is it affiliated with OpenAI in any way.

Refer to the project page and documentation for more.

Announcing mockfn

2018-03-22T19:00:00+00:00

me.pmatiello/mockfn is a library supporting mockist test-driven-development in Clojure. It is meant to be used alongside a regular testing framework such as clojure.test.

It provides two macros to be used in tests. The first, providing, replaces a function with a configured mock.

(testing "providing"
  (providing [(one-fn) :result]
    (is (= :result (one-fn)))))

The second macro, verifying, works similarly, but also defines an expectation for the number of times a call should be performed during the test.

(testing "verifying"
  (verifying [(one-fn :argument) :result (exactly 1)]
    (is (= :result (one-fn :argument)))))

Argument matchers are also supported:

(testing "argument matchers"
  (providing [(one-fn (at-least 10) (at-most 20)) 15]
    (is (= 15 (one-fn 12 18))))))

Project page

Documentation

Injecting Akka Routers as Dependencies in a Play Application

2015-07-19T18:30:00+00:00

Sometimes, an actor can’t keep up with the amount of messages it receives. When work is produced at a faster rate than it can be consumed, the system is in trouble. A bounded mailbox will drop messages that were intended to the actor while an unbounded one will grow until it consumes all the available memory and crashes the application.

If the task being performed isn’t CPU-bound or if there are enough cores available in the machine, a simple solution might be just have enough instances of this stressed actor working in parallel. Akka helps us to implement this by providing routers: actors that proxy, supervise and delegate messages to the child actors it manages. That is, when a mensage is sent the the router, it will be forwarded to one of the managed actors so that it can be properly handled. For the producer originating the messages, nothing changes: it just needs an ActorRef to the router and it can be completely ignorant about the size of the pool of consumers and whether it is pushing work directly to the consumer or through an intermediate.

The problem, then, becomes a matter of providing such a reference to the producer. Since version 2.4, Play Framework provides out-of-the-box support for Dependency Injection, which also can be used to build and inject actors where they are needed. Here is a brief example:

class StressedActor @Inject() (dependencies...) extends Actor {
  override def receive = ...
}

class StressedActorModule extends AbstractModule with AkkaGuiceSupport {
  def configure() = {
    bindActor[StressedActor]("stressed-actor")
  }
}

class Producer @Inject() (@Named("stressed-actor") talkingActor: ActorRef) {
  ...
}

// Add the following line to application.conf:
play.modules.enabled+="path.to.StressedActorModule"

With as little boilerplate as this, Play will be able to build StressedActor (along with all its dependencies) and have it injected in Producer. At the present situation, though, this isn’t sufficient as more than a single instance of the actor is demanded. Fortunately, only a small tweak at the module is needed to have a number of instances of StressedActor built under a router and to have this router injected into the producers:

class StressedActorModule extends AbstractModule with AkkaGuiceSupport {
  def configure() = {
    bindActor[StressedActor]("stressed-actor", RoundRobinPool(5).props)
  }
}

Here, a pool five consumers will be instantiated and the messages sent to the router will be conveniently delivered to these consumers in round-robin fashion such that the load on them is kept balanced.

Request-Scoped Dependency-Injection in Play Framework with MacWire

2015-03-03T09:00:00+00:00

Update (09 Jul 2015): This post was written for Play 2.3.x. Since version 2.4.0, dependency injection is supported out of the box in Play through Guice. Also, changes in routing at this version of the framework break the approach described here.

Controllers in Play Framework are usually defined as singleton objects. In fact, Play’s documentation defines a controller as a singleton object that generates Action values and provides an example like the one below:

object Application extends Controller {
  def index = Action {
    Ok("It works!")
  }
}

This kind of design is not without problems. Any dependencies of these controllers must be constructed inside the controller, tightening the coupling between them as the controller must now concern itself with one more aspect of its dependencies, namely, their construction.

This increased coupling also make the reuse of these controllers harder as it’s not trivial to tweak them by substituting their dependencies with different implementations. This is particularly relevant when writing unit tests as it’s often very interesting to substitute dependencies for mock objects.

The lifecyle of the dependencies may also become a more present concern: if a dependency is kept as an instance attribute, any mutable state in it will be shared by other requests to this controller, which may be particularly unpleasant in a concurrent environment. Avoiding this requires building the dependency independently at every action where it’s necessary.

A better design

This said, all these problems can be solved by adopting a different design. Instead of using the same singleton object for handling every request, we can have an exclusive controller instance for each request. Instead of having all dependencies statically bound to the controller, we can have them injected in runtime into the controller through its constructor.

Even though Play provides no native support for Dependency Injection, it allows us to take over the job of instantiating and providing the controller objects that will be used to handle the incoming requests. And since we can initialize these objects ourselves, we can also provide them their dependencies while we’re at it. The approach we’ll be using here is outlined in the framework documentation, even though no specifics are given on how to perform the actual initializations and injections.

Instantiating controllers manually

Consider a Play routes file with the following entry:

GET         /action                       controllers.Application.action

Any request to /action will be automatically dispatched by Play to the action method in the Application singleton object. However, if we prefix the route target with an @ character, the method getControllerInstance in the Global object (documentation) will be called to produce an instance of the Application class.

GET         /action                       @controllers.Application.action

The following snippet contains a trivial implementation for the Global#getControllerInstance method. It just produces a new instance of the given class. In fact, you don’t actually need to implement it as Global will inherit the exact same implementation from the GlobalSettings superclass.

object Global extends GlobalSettings {
  override def getControllerInstance[A](controllerClass: Class[A]): A = {
    controllerClass.newInstance();
  }
}

Notice that it takes a class as argument, not a singleton object. And the return of this method will be an instance of the given class. Therefore, our controller cannot be a singleton object as it must be a class instead.

class Application extends Controller {
  def action = Action {
    Ok("It works!")
  }
}

Introducing MacWire

MacWire is an interesting library for performing Dependency Injection in Scala applications. Like an usual DI container, it frees the developer from the tedious wiring of instances and dependencies. Unlike such containers, though, it generates all this wiring code in compile-time, backed by Scala macros. This way, if some class cannot be initialized because, say, some dependency isn’t registered, a compilation error will be raised instead of a runtime error.

The library expects the managed classes to be registered as attributes in a class, trait or object.

class AuthenticationService
class Application(authenticationService: AuthenticationService) extends Controller

class Wiring {
  lazy val authenticationService = wire[AuthenticationService]
  lazy val application = wire[Application]
}

Fetching a perfectly wired instance with a static lookup is trivial:

val wiring = new Wiring()

wiring.application

In our case, however, a dynamic lookup will be necessary. MacWire makes this easy too:

val wiring = new Wiring()
val appClass = classOf[Application]

wiredInModule(wiring).lookupSingleOrThrow(appClass)

Putting it all together

Finally, if we have a Wiring class such as the one described above, we simply have modify our Global#getControllerInstance method to use it when building our controller instances:

object Global extends GlobalSettings {
  override def getControllerInstance[A](controllerClass: Class[A]): A = {
    wiredInModule(new Wiring).lookupSingleOrThrow(controllerClass)
  }
}

Notice that we’re building a new instance of Wiring at every request. This will ensure that the returned controller and its dependencies will all be scoped to a single request. Unnecessary components won’t be constructed as long as they are registered in lazy val attributes.

Database Isolation for Tests in Play Framework

2015-01-03T19:00:00+00:00

Play Framework provides decent support for unit and functional tests. Tests using the WithApplication, WithServer or the WithBrowser abstract classes will bootstrap your application so that configurations are loaded and databases connections are available. Yet, nothing is provided out of the box to ensure that database changes made during the execution of a test will not leak into other tests or into the development environment.

About Play, Slick and specs2

Here, we’ll be using Slick for persistence and specs2 as testing framework, but the general idea should apply to different libraries without any major changes. The use of these libraries with Play is outside the scope of this post, but these links should do a good job at introducing the subject:

Isolating the test database from the development database

The first thing to do is to make our tests run in a different database than the one we use for developing the application. This will ensure that changes to the data in the database performed during the execution of the tests do not disturb the data necessary for developing the application.

In a previous post, I’ve described one possible approach for that. It envolves using a surrogate configuration file, with a different database configuration, and configuring SBT to instruct Play to load this file instead of the regular application.conf when executing tests.

Isolating tests from each other

Once that we have distinct databases for testing and development, it’s time to isolate the tests from each other so that the success or failure of them do not depend on which tests were run and in which order.

The idea here is to have a DatabaseIsolation trait that can be composed with the WithApplication, WithServer and WithBrowser abstract classes. It should empty every table in the database before each test is run. Here’s what a test should look like when this trait is used:

"do stuff using the database" in new WithApplication with DatabaseIsolation {
  DB.withTransaction { implicit session =>
    ...
  }
}

This trait can be implemented as an around scope. Notice the prepareDatabase method which is responsible for clearing the database. Its implementation will, of course, vary depending of the project and persistence solution in use.

trait DatabaseIsolation extends Around with Scope {
  abstract override def around[T: AsResult](t: => T): Result = {
    super.around {
      import play.api.Play.current // Put implicit Play application in scope
      DB.withTransaction { implicit session => prepareDatabase }
      t // Execute test
    }
  }

  protected def prepareDatabase(implicit session: Session): Unit = {
    (Schema.someTable :: Schema.otherTable :: Nil).foreach(_.delete)
  }
}

In tests, Play’s default behaviour is to apply the database evolutions automatically when necessary, so we don’t have to worry about that here.

Seeding the database with test data

At some tests, we may want to have a minimal database with some predefined data instead of an empty one. Given a Seed object capable of filling the database with such data, we can define a SeededDatabase trait as a child trait of DatabaseIsolation, described above. It should be able to, before each test, empty the database just like before but also add some data to it leaving the tables in a predictable state.

trait SeededDatabase extends DatabaseIsolation {
  abstract override protected def prepareDatabase(implicit session: Session): Unit = {
    super.prepareDatabase
    Seed.seed // Fill the database with some test data
  }
}

This trait can be used like the other:

"do stuff using the database" in new WithApplication with SeededDatabase {
  DB.withTransaction { implicit session =>
    ...
  }
}

Conclusion

Proper database isolation for tests is important to avoid spurious successes (which hide problems in the tested code) or failures (which reduce our trust in the tests). It also helps us to avoid seemingly non-deterministic behaviour in our tests and provides us with a predictable, clean state for our tests to execute upon, making them simpler to reason about.

Play provides only basic database support (configuration, connection pools, evolutions) and it doesn’t prescribe any specific persistence library. Although this provides a lot of flexibility, it also means that the framework can’t do much to isolate the database during the tests. Yet, as we can see, it’s possible to implement such isolation very easily, cleanly and without any changes to the production code.

Using a different configuration file when running tests in a Play project

2014-12-26T11:00:00+00:00

Play Framework, as of version 2.3.x, typically loads the configuration for your application from conf/application.conf. Although it works as expected, it doesn’t allow for different configurations to be used in different environments unless the file is replaced. This is specially inconvenient when running tests directly from Activator since these tests will be executed against the same database used for running the application in development mode.

Luckily, Typesafe Config and SBT provide all that is necessary to work around this issue cleanly.

First create a configuration file, say conf/application.test.conf, for the Test mode. This file will inherit all settings defined in the main configuration file through the include statement, but will also override the values that should be different when running tests.

include "application.conf"

db.default.driver="org.postgresql.Driver"
db.default.url="jdbc:postgresql://localhost/project_name_test"
db.default.user="username"
db.default.password="password"

Then, configure SBT to instruct Play to load this file when running in Test mode by adding the line below to your build.sbt.

javaOptions in Test += "-Dconfig.file=conf/application.test.conf"

This should be enough.

Futures in Scala

2013-10-28T21:00:00+00:00

Scala 2.10 introduced futures as a convenient abstraction for concurrent programming. Using futures, one can perform a number of computations in parallel for which the result is expected to be available, at some point, in these Future objects.

A result in a Future can be easily retrieved without blocking the execution flow by setting a callback to be invoked once it’s ready:

val fut = future { slowComputation }

fut.onSuccess {
	case result => useSuccess(result)
}

This callback is guaranteed to be executed after the future completes successfully. It’s also possible request a callback to be executed if the future fails with an exception:

fut.onFailure {
	case exception => useError(exception)
}

Alternativelly, a single callback can be defined for handling both success and failure cases:

fut.onComplete {
	case Success(result) => useSuccess(result)
	case Failure(exception) => useError(exception)
}

Finally, if blocking is acceptable, one can simply wait until the value is available:

Await.result(fut, Duration.Inf)

This isn’t what you probably want, though.

Some convenient operations

Even though futures are easier to handle than most other approaches to concurrency, they are not really a transparent solution. In fact, sometimes they can be a little tricky to use while still retaining its advantages. For instance, how does one, holding a future for some value, obtains a modified version of this value without waiting for its evaluation?

Fortunatelly, the API for futures in Scala is quite helpful in cases like this.

From future of value to future of modified value

Given a future fut for a value v and a function f, it’s quite straightforward to obtain a new future for a value f(v) without waiting for v to be evaluated:

fut.map(f)

From list of futures to future of list

It’s possible turn a list of futures into a future of a list for the same values. That is, given a List of Future objects List(Future(v1), Future(v2), ...), it is possible to produce a Future object such as Future(List(v1, v2, ...)):

Future.sequence(list)

Reduce list of futures to future value

It’s also possible to take a list of futures List(Future(v1), Future(v2), ...) and a reduction function f to produce a new future containing List(v1, v2, ...).reduce(f) as value:

Future.reduce(list)(f)

Reduce future of list to future value

Finally, given the future of a list Future(List(v1, v2, ...)) and a reduction function f, a new future containing List(v1, v2, ...).reduce(f) as value can be produced:

fut.map { list => list.reduce(f) }

ScalaCheck (slides)

2013-08-28T12:00:00+00:00

Presented at Scaladores, August/2013.

A few tricks for writing faster Ruby code

2013-07-09T12:00:00+00:00

Recently, I was busy at work trying to make some Ruby code we’ve had written run faster. This is somewhat outside my zone of confort: I’m mostly concerned about design, correctness, testability, etc, and most of the time, my concerns about performance are restricted to proper choice of algorithms, data structures and caching strategies. Although I’ve done my share of profiling and performance improvements on Java applications, this was my first time doing something like that in Ruby. And as I haven’t found much on this subject on Google, I decided to share here some of the stuff that proved themselves helpful to my problem.

The benchmark module

The benchmark module, bundled with Ruby standard library, is an useful tool for deciding among candidate implementations based on performance. Basically, the module “provides methods to measure and report the time used to execute Ruby code” (source), which is what you’d want in this situation.

The Benchmark::bm method, particularly, is quite convenient. You can setup a few different implementations, have them run, and then have their execution times reported. An example follows:

Benchmark.bm do |x|
  x.report "Candidate #1" do
    candiate_one
  end
  x.report "Candidate #2" do
    candidate_two
  end
end

The produced output is something along the following format:

               user       system     total      real
Candidate #1   6.260000   0.000000   6.260000   (6.260650)
Candidate #2   3.290000   0.000000   3.290000   (3.292514)

This kind of measuring is unavoidable. First, attempts to improve the performance of a code may have the opposite effect, making it perform worse. Second, even tricks known to work well under a specific version of Ruby won’t necessarily have the same effect on a different one.

Caveat emptor: All benchmarks here were executed under Ruby 1.9.3p385 on a Early 2011 MacBook Pro running OS X Mountain Lion. You may have different results when running these examples on a different environment.

Embracing mutability

Immutability is, for many good reasons, considered a good practice. It usually results in cleaner, safer, more elegant code. It avoids a number of problems on concurrent programs. Yet, unfortunatelly, there may be a performance cost in avoiding mutable state. We can observe this by comparing the performance of String#downcase and String#downcase! against a list of 10 millions different strings. Both methods turn all letters of the original string into lowercase letters. The former, though, produces a new string and leaves the original one unchaged. The later performs the operation in place, modifying the original string.

list = (1..10000000).map { |i| "Lorem ipsum dolor sit amet. #{i}" }
Benchmark.bm do |x|
  x.report "String#downcase" do
    list.each { |str| str.downcase  }
  end
  x.report "String#downcase!" do
    list.each { |str| str.downcase! }
  end
end

                   user        system     total       real
String#downcase    10.970000   0.430000   11.400000   (11.393009)
String#downcase!   5.340000    0.040000   5.380000    ( 5.377566)

Clearly, the mutable version is significantly faster — probably because it lacks the overhead of object construction and memory allocation.

Accepting different semantics

Sometimes, some extra performance can be achieved by using solutions that offer a different, yet still similar, semantics. We can take, for instance, the Hash#fetch method. This method kindly retrieves from a hash the value assigned to the given key. If no value is assigned, the method returns a specified fallback value.

>> {:a => 1, :b => 2}.fetch(:a, 0)
=> 1
>> {:a => 1, :b => 2}.fetch(:c, 0)
=> 0

There is a simple alternative solution, using Hash#[], with similar semantics, though:

>> {:a => 1, :b => 2}[:a] || 0
=> 1
>> {:a => 1, :b => 2}[:c] || 0
=> 0

In fact, the hash[key] || fallback trick is indeed somewhat faster, as this benchmark shows:

hash = {:a => 1, :b => 2, :c => 3}
Benchmark.bm do |x|
  x.report "fetch" do
    50000000.times { hash.fetch(:a, 0) }
  end
  x.report "[] ||" do
    50000000.times { hash[:a] || 0 }
  end
end

        user       system     total      real
fetch   8.340000   0.000000   8.340000   (8.339169)
[] ||   5.100000   0.000000   5.100000   (5.100304)

It’s worth remembering, though, that even though the semantics of these two solutions are similar, in some cases they produce different results:

>> {:key => false}.fetch(:key, true)
=> false
>> {:key => false}[:key] || true
=> true

I suppose that a decision regarding the use this optimization requires both careful measuring and reflection on whether the performance improvement is worth the risks involved.

Memoization

Memoization is an old and well-know technique to improve the performance of a program by storing the result of function calls so that repeated calls to the same function with previously processed parameters can just be fetched from memory instead of being recalculated.

A classical example is the algorithm for producing the Fibonacci sequence:

The mathematical definition for the Fn sequence of Fibonacci numbers is:

F(n) = F(n-1) + F(n-2), for n > 1
F(1) = 1
F(0) = 0

This definition has a very straightforward translation to Ruby:

def fibonacci(n)
  n <= 1 ? n : fibonacci(n-1) + fibonacci(n-2)
end

This is cute, but really slow. A calculation of fibonacci(5), for instance, would involve the following calls:

fibonacci(5) = fibonacci(4) + fibonacci(3)
	fibonacci(4) = fibonacci(3) + fibonacci(2)
		fibonacci(3) = fibonacci(2) + fibonacci(1)
			fibonacci(2) = fibonacci(1) + fibonacci(0)
				fibonacci(1) = 1
				fibonacci(0) = 0
	fibonacci(3) = fibonacci(2) + fibonacci(1)
		fibonacci(2) = fibonacci(1) + fibonacci(0)
			fibonacci(1) = 1
			fibonacci(0) = 0

Here, we have a number of repeated calls. This isn’t so bad for n = 5, but this effect is much, much worse for higher values of n. We can avoid all this repetition with a memoized version of the same algorithm:

FIBONACCI_CACHE = {}
def memoized_fibonacci(n)
  FIBONACCI_CACHE[n] ||= (n <= 1 ? n : memoized_fibonacci(n-1) + memoized_fibonacci(n-2))
end

Here are the running times for the original and memoized versions for n = 40:

           user        system     total       real
original   24.210000   0.010000   24.220000   (24.215359)
memoized   0.000000    0.000000   0.000000    (0.000030)

Things were a bit more complicated in the problem I was working on, though. The method I had to memoize returned Hash instances that were modified by the caller method. This is obviously problematic as a second call would receive a dirty, already modified, object instead of a good one. This was easily solved by returning a defensive copy of the memoized hash. Although Object#clone added some overhead, it was still much faster than recalculating the whole thing once again.

Symbol#to_proc

Most of the time, in order to make some code run faster, we also make it uglier. This is kind of expected: if the beautiful solution is also fast, you’re very likely already using it. This isn’t, of course, true all the time. For some reason, I was expecting the cute Symbol#to_proc trick to perform poorly, so I’ve tried changing it to explicit blocks. This proved to be a bad idea:

Benchmark.bm do |x|
  x.report "to_proc" do
    (1..100000000).each &:to_s
  end
  x.report "block" do
    (1..100000000).each { |n| n.to_s }
  end
end

          user        system     total       real
to_proc   24.410000   0.010000   24.420000   (24.421814)
block     28.640000   0.010000   28.650000   (28.656124)

A Kind of Informal Introduction to π-Calculus

2013-01-06T12:00:00+00:00

The π-Calculus is one of many approaches to concurrent computation by the means of formal modeling. Its purpose is to enable us to reason about concurrent processes in a disciplined fashion by manipulating expressions through formally defined algebraic rules.

Any exposition of the calculus will eventually introduce you to the primitive notions:

Agents, which can be understood and even referred as processes.
Actions, which are anything that can be done by an agent.
Channels, which are links connecting agents, alowing them to communicate.
Names, which are exchanged through channels.

The general idea is that we have a set of agents connected to each other through channels in some sort of network. These agents then use these channels communicate to each other by exchanging names.

The role of communication is central to the π-Calculus. In fact, there are only three possible actions for an agent to perform, and two of them regard communication:

The Output action x̅a: The name a is sent through the channel x.
The Input action x(a): The name a is bound to whatever is received through channel x.
The Silent action τ: Any action internal to the agent (i.e. involving no communication).

Now that we have actions, we can already build some agents:

The Nil, empty agent 0.
The Prefix α.P that executes the action α and then continues as the agent P.
The Restriction (νx)P that behaves as P having the name x in its local scope.
The Parallel Composition P|Q that behaves as agents P and Q executing in parallel.
The Sum P+Q that can behave either as P or Q (but not both).
The Match [x=y].P that behaves as P if x and y are the same name.
The Mismatch [x≠y].P that behaves as P if x and y are not the same name.

This may be a bit too abstract, but a small example will hopefully help: suppose we have a printer P, a server S and a client C. We would like to have a name sent by the client printed by the printer. We also define that the server is connected to the client by a channel y and to the printer by a channel x.

To achieve our goal, C should send a name to S which, in turn, should forward it to P. We can write:

C|S|P, where

C=(νa)y̅a
S=y(b).x̅b.S
P=x(c).τ.P

Notice two things:

First, although the client agent terminates after sending its message, the server and printer agents return to their initial state.
Second, all communication in the π-Calculus is synchronous, so an agent sending a message will remain blocked until some other agent receives the message. Also, an agent listening on a channel for a name will remain blocked until a message is actually sent through this channel.

At this point, we’re ready to talk about what makes the calculus really powerful: in π-Calculus, all channels are names, which means that they can be sent through other channels just like any ordinary name. That’s really important: an agent receiving a name that is also a channel may now use this received channel to communicate with other agents. Essentially, such interactions modify the network between the agents in the computation. This is called dynamic reconfiguration.

We can illustrate this phenomenon by modifying our previous example. Previously, we had the client sending the message to the printer through the server. Now, we’ll have the client to send the message directly to the printer through a channel provided by the server. We’ll write:

C|S|P, where

C=(νa)y(z).z̅a
S=y̅(x).S
P=x(c).τ.P

And that’s enough for now. A lot more can be found in the book written by Robin Milner, the main author of π-Calculus. I also find Parrow’s paper very readable.

Lazy Evaluation em Scala (slides, pt-br)

2012-07-07T12:00:00+00:00

Presented at: TDC 2012, July/2012.

Por que dizemos que Scala é uma linguagem funcional? (slides, pt-br)

2012-06-13T12:00:00+00:00

Presented at: VII Scaladores, June/2012.

Implementing interface contracts in Python with class decorators

2012-03-23T12:00:00+00:00

Python decorators are quite useful and interesting. I’ve already written about function decorators in a previous post and class decorators are a worthy followup.

I order to illustrate this feature, we’ll implement support for interface contracts for Python classes. In a language like Java, for instance, a contract can be declared this way:

public interface Comparable<T> {
    public int compareTo(T o);
}

And a class declared as an implementation of this interface must provide the relevant methods (with the correct signature). Otherwise, a compilation error arises. For instance:

public final class Integer extends Number implements Comparable<Integer> {

    ...

    public int compareTo(Integer anotherInteger) {
        int thisVal = this.value;
        int anotherVal = anotherInteger.value;
        return (thisVal&lt;anotherVal ? -1 : (thisVal==anotherVal ? 0 : 1));
    }

    ...

}

We’ll try to achieve a similar feature in Python, albeit with some limitations. First, as Python does not have an explicit compilation step, our errors will only arise in runtime. Second, in Python, the types of the arguments and return values aren’t present in the signature of methods, so we’ll leave it out too. This is a notable absence, but we can still verify the presence of desired methods in the implementer type and the following elements in the method’s signature:

Name
Number of arguments
Name of arguments (as Python supports named parameters)
Default values for arguments (as Python supports default arguments)
Support for variadic functions (indefinite number of arguments)

This said, we now have to declare our interfaces. I picked the following (somewhat cumbersome) syntax:

class some_interface(interface):
    simple_method = method(['self'])
    method_with_args = method(['self', 'arg1', 'arg2'])
    method_with_varargs = method(['self'], varargs='varargs')
    method_with_kwargs = method(['self'], keywords='kwargs')
    method_with_default_argument = method(['self', 'default_arg'], defaults=1)

Here we have an ordinary class subtyping an interface class. The methods are defined as attributes of the some_interface class, holding instances of the method class.

The interface class isn’t really exciting or even necessary. Actually, it exists for the sole purpose of tagging an interface contract as such:

class interface(object):
    pass

The method class holds the method signature.

class method(object):
    def __init__(self, args=None, varargs=None, keywords=None, defaults=0):
        self.args = args or []
        self.varargs = varargs
        self.keywords = keywords
        self.defaults = defaults

Notice that the constructor takes a few arguments:

`args` — the list of arguments
`varargs` — the name of the list holding extra arguments, if any
`kwargs` — the name of the dictionary holding extra named arguments, if any
`defaults` — the number of arguments with a default value

Now we can create a new type implementing some_interface. And it follows:

@implements(some_interface)
class some_class(object):

    def simple_method(self):
        pass

    def method_with_args(self, arg):
        pass

    def method_with_varargs(self, *varargs):
        pass

    def method_with_kwargs(self, **kwargs):
        pass

    def method_with_default_argument(self, default_arg=100):
        pass

Notice that we declare the class as implementing the contract through the implements class decorator.

First thing we have to observe is the initialization of the implements decorator. Once the decorated class is loaded by the interpreter, an instance of the decorator is produced. The argument some_interface is passed to the constructor during the initialization (and it’s stored as an attribute for future reference).

class implements(object):

    def __init__(self, interface):
        self.interface = interface

    ...

Then, the __call__ method of the decorator is invoked. The decorated class (some_class in this example) is passed as argument. In this method, it is verified whether the given class satisfies all of its contractual obligations. If no problem arises, the class is returned at the end of the call.

class implements(object):

    ...

    def __call__(self, clazz):
        methods = [each for each in dir(self.interface) if self._is_method(each)]
        for each in methods:
            self._assert_implements(clazz, each)
        return clazz

    ...

Notice that the _assert_implements method is called against each method declared in the interface. This method, listed ahead, verifies whether the given method, as implemented by the class, has the desired signature, as specified in the interface. In case it doesn’t, an exception is raised (effectively interrupting the program).

class implements(object):

    ...

    def _assert_implements(self, clazz, method_name):
        method_contract = object.__getattribute__(self.interface, method_name)
        method_impl = getargspec(object.__getattribute__(clazz, method_name))
        assert method_name in dir(clazz)
        assert method_contract.args == method_impl.args
        assert method_contract.varargs == method_impl.varargs
        assert method_contract.keywords == method_impl.keywords
        if (method_impl.defaults is not None):
            assert method_contract.defaults == len(method_impl.defaults)
        else:
            assert method_contract.defaults == 0

    ...

The object.__getattribute__() (documentation) call is one thing worth of note as it is part of Python’s introspection features and targeted at retrieving attributes from classes and objects. Here, it’s used both to retrieve method objects from the interface and and to obtain references to the actual methods in the concrete class (the clazz argument).

The getargspec function (documentation) is also an interesting introspection feature. Given a function, it will return an object with information about the given function’s signature.

So, having both the expected signature from the method object hosted in the interface and the signature of the method implemented in the concrete class, we can simply compare them both and hope for a match.

(Postscript: although the source code presented above works fine, it’s actually a shorter version of the code originally developed for the purpose and available here.)

Pistache: A π-Calculus Internal Domain Specific Language for Scala (slides)

2011-09-25T12:00:00+00:00

Presented at: XIV Simpósio Brasileiro de Métodos Formais (SBMF 2011), September/2011.

Dependency Injection in Ruby as inspired by Scala

2011-03-15T12:00:00+00:00

A recent discussion in the GOOS’ group has lead me to consider different ways to compose objects in Ruby. Specifically, as module inclusion seems to be the favored approach for adding stuff to classes in Ruby, I’ve became interested in finding a more flexible idiom for this.

The objective, therefore, is to define an instance variable in a module and be able to have it injected in instances of some class. Since I’m not that familiar with Ruby yet, I’m forced to turn to other languages for inspiration. A possible solution in Scala, presented below, is kind of intuitive.

Let’s start defining a simple trait Lightsource.

trait Lightsource {
  def on
  def off
}

The goal is to have a instance of a class implementing this trait injected in every instance of the class Room below.

abstract class Room {
  val lightsource:Lightsource
  def enter = lightsource.on
  def leave = lightsource.off
}

The injection can be performed by mixing-in the Room class with traits designed to provide a Lightsource. For example:

trait FluorescentLamp {
  val lightsource = new Lightsource {
    def on = println("Sad light on")
    def off = println("Sad light off")
  }
}

The actual composition reads nicely:

val sadRoom = new Room with FluorescentLamp

If the fluorescent lamp is unsatisfactory, an alternative implementation can be provided:

trait IncandescentLamp {
  val lightsource = new Lightsource {
    def on = println("Warm light on")
    def off = println("Warm light off")
  }
}

val warmRoom = new Room with IncandescentLamp

And I will stop at this point. Although there is opportunity for improvement, the implementation is satisfactory enough for me.

Having succeeded at the Scala front, we can tackle the same issue using Ruby. Here, we find ourselves both unable and not required to declare interfaces, so we move directly to the definition of our Room class.

class Room

  def enter
    @lightsource.on
  end

  def leave
    @lightsource.off
  end

end

The next step is to define a module responsible for providing an @lightsource object to instances of the Room` class. For instance:

module FluorescentLamp
  class FluorescentLampImpl
    def on
      puts "Sad light on"
    end

    def off
      puts "Sad light off"
    end
  end

  def initialize(*args, &b)
    super
    @lightsource = FluorescentLampImpl.new
  end
end

Alternatively:

module IncandescentLamp
  class IncandescentLampImpl
    def on
      puts "Warm light on"
    end

    def off
      puts "Warm light off"
    end
  end

  def initialize(*args, &b)
    super
    @lightsource = IncandescentLampImpl.new
  end
end

The actual composition does not read so nicely, but surely it can be improved by some sort of DSL. Being lazy, I’ll skip that, though:

sadRoom = Class.new(Room) do
  include FluorescentLamp
end.new

warmRoom = Class.new(Room) do
  include IncandescentLamp
end.new

Now, I’m suppose that this approach is not really within Ruby’s orthodoxy, but I found it interesting nevertheless. Also, it surprised me that the Scala version is both cleaner and smaller than the Ruby version, while still providing the safety advantages of the static typing.

In which I give my own half-baked workaround to the lack of tail call optimization in Python

2010-09-22T12:00:00+00:00

A tail call is a function call such that it is the last action performed by a procedure. Therefore, the value returned by the caller procedure is the value returned by the called procedure. Many compilers and interpreters take advantage of this situation by using the caller’s stack space to execute the called procedure, instead of allocating more space for it. Because no extra space is consumed by these calls, recursive tail calls can be nested at will without risk of overflowing the stack.

Unfortunately, Python’s interpreter does not perform this optimization trick. This can be easily learned from the example bellow:

>>> def f(x):
...     if (x>0): return f(x-1)
...     else: return 0
...
>>> f(10)
0
>>> f(1000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
  ...
  File "<stdin>", line 2, in f
RuntimeError: maximum recursion depth exceeded

This limitation in the interpreter is unlikely to disappear anytime soon, if at all. But, as tail call optimization is often very convenient, many workarounds were written. These workarounds, with different levels of success, make use of decorators (which are functions that are executed before the decorated function) to control the execution of function calls.

This post presents yet another solution, also based on decorators. The objective here is to avoid any limit on the number of recursive tail calls performed by a function by decorating it like bellow.

@tail_recursive
def f(x):
    if (x>0): return f(x-1)
    else: return 0

Every call to f, then, will be intercepted by the decorator. Depending on the situation, it may continue and execute the function as usual, or return an object that may be used to perform the issued call later. The next listing presents the class for these objects (note that they store both a reference to a function and the arguments for a specific call).

class _continue(object):

    def __init__(self, func, *args, **kwargs):
        self.func = func
        self.args = args
        self.kwargs = kwargs

Each call to a decorated function f implies on a independent run of the decorator code. On the first call, the decorator will allow f to be executed, but instead of returning immediately, it will await for _continue objects to execute them. Subsequent tail-recursive calls to f, on the other hand, will just return an instance of _continue filled with the call arguments without executing f. Non tail-recursive are allowed to proceed unmolested.

This approach keeps the stack from growing by avoiding nested calls whenever possible. Still, detecting recursion and tail recursion are tricky jobs. The former is performed by checking if penultimate frame in the stack refers to our decorator. The later is performed by checking whether the bytecode associated with the previous stack frame will return immediately after the completion of the call.

The source code for the decorator follows. An instance of it is produced for each decorated function, and the method __call__ is executed on every invocation of them.

OPCODES = [chr(opmap['RETURN_VALUE']), chr(opmap['POP_TOP'])]

class tail_recursive(object):

    def __init__(self, function):
        self.function = function

    def _is_tailcall(self, frame):
        caller_frame = frame.f_back
        code = caller_frame.f_code.co_code
        ip = caller_frame.f_lasti
        return code[ip + 3] in OPCODES

    def __call__(self, *args, **kwargs):

        frame = getframe()

        if frame.f_back and \
            frame.f_back.f_back and \
            frame.f_back.f_back.f_code == frame.f_code and \
            self._is_tailcall(frame):
            return _continue(self.function, *args, **kwargs)

        retval = self.function(*args, **kwargs)
        while (True):
            if (type(retval) == _continue):
                retval = retval.func(*retval.args, **retval.kwargs)
            else:
                return retval

(The entire code, with tests, is available here.)

Of course, this isn’t true tail call optimization, but only a trick to achieve unlimited tail recursion. Properly implemented, the optimization should also provide increased performance by reducing the work needed to perform a function call; here, performance is actually harmed.