Surya Suravarapu's Blog

Ramblings on software and stuff related

Posts Tagged Lift

Conditional Requests with Lift

Jan 13

Posted by Surya Suravarapu in REST | 4 Comments

ETag (or entity tag) values and/or Last modified time of the resource are typically used for this purpose. I'm only discussing ETags here, interchanging this with Last-modified time is trivial, so skipping.

In this post I'm concentrating on deep ETags, where application developer can generate and compare ETags based on the underlying domain objects, database tables, etc.

Other kind of ETags, the shallow ones, can be supported at the framework level. They rely on hash of the representations. A web framework can generate ETag value, and compare them with the representation from the response. Shallow ETags are useful with respect to saving bandwidth but does not eliminate the computation on the server side. (Expect a post on the shallow ETags soon).

Conditional GET

Conditional GET is a great way to conserve bandwidth. An intermediary cache may check with the origin server whether the resource has changed since it last received a representation. The server responds either with the new representation if the resource state changed or send back only the headers with 304 Not Modified response.

Let's start with defining a Product class which is using Lift's Mapper (as ORM). Also, note the use of CreatedUpdated trait, this will automatically add two timestamp fields -- createdAt and updatedAt for insert and update operations respectively.

There are various strategies to generate ETags, I'm using the one that uses the updatedAt field (and use its Long value). Let's first see this in action and get back to implementation details in a bit. Using cURL to test.

Request and Response for a Product of known ID

For subsequent requests the client sends the value of ETag provided by the server. See If-None-Match header in the request below. Adding this header makes the request a conditional one. If the resource doesn't change the server sends back only the headers with 304 header (see below).

As far as implementation is concerned, relevant portion of the code is provided below:


Value of If-None-Match header from the request is compared with the resource ETag value and then either respond with 304 (resource not modified) response or 200 (ok) response. Note that the value of If-None-Match can be an array of ETag values separated by commas, which is accounted for in the code above. NotModifiedResponse used above can very well be a standard sub class of LiftResponse in the framework. Regardless, you could create one as follows, which is actually a wrapper around Lift's InMemoryResponse

Conditional PUT

Conditional PUT is a great approach to enforce that the client is updating the most recent version of the resource state. Client does a GET first and gets the ETag value and uses that in the If-Match header (see below).  The usage of If-Match makes it a conditional request for updates. Server can enforce this by rejecting any updates without If-Match header in the request.

If the ETags match the resource state is updated. The server responds back with 204 (No Content) and with the new ETag value.

Suppose some other client that doesn't have the updated ETag value tries to send a request to update. The server responds with 412 (Precondition Failed) with the new ETag header value (shown below)

Implementation-wise, the code below compares the ETags and responds with either 204 or 412 indicating success or failure of conditional update (It also checks the request's content type and the existence of the resource and respond appropriately).

Just like in the case of GET, added NoContentResponse and PreConditionFailedResponse, both are wrappers around InMemoryResponse.

Complete source of the service is here, just in case.

Tags: , ,

JavaOne 2010 – My Impressions (Part 2)

Sep 30

Posted by Surya Suravarapu in General | 3 Comments

Part 1 of the series is here.

Sessions (Contd.)

Building Real-Time Web Applications with Lift

David Pollak, founder of Lift (an open source) Scala-based web framework, did a live demo building a chat application. I haven't seen many sessions (that I attended) in the conference where the presenters did code demos. David pulled it off without major glitches. He developed a comet-based chat application along the way touching the concepts of Lift's templates, snippets and Ajax support.

After the demo he went over some of the features of Scala and Lift. Lift applications are secure by default, and he mentioned that penetration testers are having tough time finding security vulnerabilities in the application. Some of the factors that makes this secure: Lift's forms mechanism associates randomly generated GUIDs with form elements and functions. Risk of replay attacks is relatively low with Lift applications. Lift builds all web pages as well formed XHTML rather than a String or a stream of characters or bytes. Lift's SiteMap provides unified menu generation and access control. All these and more actually makes the developers life lot easy by having them concentrate on the business logic of the domain and not on the underlying plumbing work.

Performance concerns with one of the applications built on Rails caused Pollak to think about a Scala-based web application utilizing the power of the JVM (which eventually improved the performance). Lift borrows a lot of good ideas from Rails and other frameworks into Lift.

Foursquare and Novell Pulse are mentioned as publicly available apps that are built using Lift. There are many more startups and enterprise applications building interactive applications using the framework.

Mint.com's Technology Behind the Scenes: Practical Lessons for Scalable Web Apps

David Michaels and Daryl Puryear presented what turned out be one of my favorite sessions of the conference. Mint.com is world's largest free personal financial application with four million users covering 40K zip codes and all 50 states of the USA. Currently hosts 20 millions financial accounts with over 15K financial institutions.

The speakers went over step-by-step how they built the foundation of quality and security, and went over each of the other aspects of the pyramid: performance, scalability, manageability and maintainability. They traveled back in the time and shared the excitement and environment at the launch time some time in 2006/7 time frame. During that time they have a well-defined feature set but with unknown traffic. They wisely invested in production performance monitoring. Doing this enabled Mint to be proactive and enable fast triage of the issues. Monitored only the expensive operations to keep the overhead low. They decided to build a monitoring application (rather than buying) as they have some in-house expertise. Used Spring's auto-proxy feature. Aggregated results in memory and persist periodically. Simple interface was built on top to analyze the data.

Mint performed some performance tests and found that their database is the bottleneck (sounds familiar?). So they tuned the application code to reduce database usage. That brought only marginal improvement, and they hired services of an outside DB conultant who helped optimized the MySQL configuration, which ultimately removed the bottleneck. Lesson: small team can't have all expertise, hire consultants occasionally.

Traffic continued to grow, there were major spikes after press coverage. Once again database scaling was the need of the hour. They did horizontal scaling with multiple smaller databases. Shards are based on user and they made every user independent with no refs between the users. Non-user data is separated into logical databases (user lookup, shared data, monitoring data, user data).

Mint made continuous investments into beefing up security. Standard items were covered like data encryption, penetration testing, automated security scans, multiple DMZs and secure datacenter. Mint also took care of their application architecture so that a developer mistake will not result in a security hole.

Apart from that they have demoed an application they built for triaging errors in application logs, which was interesting. Mint is using XMLC as part of their frontend! Overall this is a very informative session not just from the technology standpoint but also to hear a startup's success story.

Ninety-Seven Things Every Programmer Should Know

Kevlin Henney and Kirk Pepperdine presented one of the entertaining sessions that I attended. Kevlin Henney in particular is an excellent speaker who knows how to present the message in an entertaining way.

If you haven't read this book yet, here are those 97 things appeared in the book. Speakers chose sixteen out of the list for the talk:

Ubuntu coding for your friends - Aslam Khan; Do lots of deliberate practice - Jon Jagger; Know your IDE - Heinz Kabutz; Put the mouse down and step away from the keyboard - Burk Hufnagel; Code reviews - Mattias Karlsson; Read code - Karianne Berg; Comment only what the code cannot say - Kevlin Henney; Code in the language of the domain - Dan North; Prefer domain-specific types to primitive types - Einar Landre; The road to performance is littered with dirty code bombs - Kirk Pepperdine; Interprocess communication affects application response time - Randy Stafford; The longevity of interim solutions - Klaus Marquardt; Two wrongs can make a right (and are difficult to fix) - Allan Kelly; Testing is the engineering rigor of software development - Neal Ford; Write tests for people - Gerard Meszaros; The boy scout rule - Uncle Bob

One of my favorites is the boy scout rule interpretation for the code:

Always check a module in cleaner than when you checked it out.

This is one of those sessions that is difficult to summarize in a post like this. Checkout a video from the speaker on the same topic but at a different venue.

Simpler Scalability, Fault Tolerance, and Concurrency Through Actors and STM

Akka is a tool that you should seriously cosider if you are writing applications with high concurrency needs. Jonas Bonér, the founder of Akka, presented the concepts pertaining to its software transactional model (STM), actors and persistence mechanism. I have recently started evaluating various concurrency models, and this presentation is exactly what I needed for some concrete guidance!

Actor model was popularized by Erlang which emerged in mid-80s. Actor model is a higher level abstraction, for the developers working on concurrent applications, dealing with the traditional locking and thread management. Actors do not share state with the other actors. Each actor has a mailbox and they can only interact with other actors by sending messages. All processing is done asynchronously, actors do not block so they are excellent candidates for event-based applications. Akka actors are extremely lightweight, you can create millions of them on a single workstation. Jonas Bonér discussed the Java API, Akka has also got a Scala API.

Akka's supervision model is inspired from Erlang. It goes by the "let it crash" approach implemented by linking actors. Akka's approach is that failures do happen, don't try to prevent them as they are inevitable. Your goal should be to make them fail soon and let a supervisor (who has a bigger picture) deal with it. A supervisor is an actor responsible for start, stop and monitoring child actors. Bonér discussed All-for-one (restart all the components that a supervisor is managing) and One-for-one (restart only the crashed actor) strategies.

STM model sees the memory as the transactional dataset. It can provide atomicty, consistency and isolation attributes to a transaction. Transactions are retried automatically after the collision. Akka STM is based on the ideas of Clojure STM, a compelling idea for providing transactional shared state. Akka also has pluggable storage backend currently supports Cassandra, MongoDB and Redis.

It is one of those sessions that I felt organizers saved the best for the last (day)!

Tags: , ,

Lift and Content Negotiation

Aug 19

Posted by Surya Suravarapu in REST, Scala | No Comments

Overview

This is a follow up of my previous post on Lift and REST discussing the aspects of URI matching and content negotiation. Lift supports Accept header of HTTP by responding back with a representation in accordance with the media type provided in the header. However, it currently doesn't support quality factor (q parameter) of the Accept header, out of the box. Here  I will attempt to provide an approach to provide that support and along the way let's explore another compelling feature of the Lift's REST support.

HTTP Accept Header

Check RFC 2616 for detailed explanation on Accept and other HTTP headers. Let's go over this based on an example: If a client sends an Accept header something like the following --

Accept: application/xml; q=0.8, application/json

is interpreted as: I prefer JSON representation but if you don't have it XML is my second choice.

Quality factor or q parameter is the one that's used to specify the preference. q is a decimal ranging from 0.0 to 1.0 and is delimited with a semi-colon (;) following the media mime type. If no q parameter is specified it defaults to value of 1.0, indicating first among the options provided.

Approach

Before going into the approach for supporting the q parameter (for your Lift-based application) let's get into one of the things that you definitely want to see in a web framework: decouple business logic from the representation. Lift doesn't disappoint you in this area. In my case I was using the same business logic, authorizing the user and making the database lookups, returning the appropriate representation independent of the business logic.

serveJx in RestHelper is there precisely for that purpose. Following code provides the URI matching rule (matches /api/user/{user_id} for GET requests) and returns an object that's of trait Convertable. Also, define an implicit def that converts the object to the appropriate representation (XML or JSON, in this case).

Relevant pieces of case class User is provided below. If you are familiar with Squeryl you may have already identified the annotations provided for the constructor arguments otherwise don't worry about it; Squeryl is an excellent Scala-based ORM (I like Squeryl quite a bit, that's a topic for another blog post!). User implements Convertable, meaning the two representations -- XML and JSON via toXml and toJson functions respectively.

So with all the above in place, the following request would result in an XML or JSON response based on the Accept header. The logic for identifying the appropriate representation  is in the RestHelper's implicit def jxSel shown below. As RestHelper does not support q parameter out of the box, UserManagementService extended from RestHelper changes the behavior by overriding jxSel.

Actual Parsing: Parsing of the Accept header and determining the representation is done using functions in the ClientMediaPreference object. Instead of embedding the code in this already lengthy post, here is a link to the gist covering the parsing logic.

Conclusion

There are a couple of areas that I still have to tighten-up the code, but that's the general idea. One of the Todo items is to send a 406 Not Acceptable response if the representation that a client asks for is not implemented by the server.

Before closing, let's continue checking off another item from Tilkov's litmus test (as we did in the last post) ...

Can I easily use the same business logic while returning different content types in the response?

Answer: Yes, the framework is flexible in this aspect.

Tags: , ,

Lift and REST: URI Matching and Content Negotiation

Aug 3

Posted by Surya Suravarapu in REST, Scala | 9 Comments

As a Scala enthusiast I'm currently evaluating Lift, particularly the aspect of its REST support. This could become a series of posts on the topic as I try to understand the framework better and may be subject it to a litmus test proposed by Stefan Tilkov. Of course, I will not be making a judgment call whether the framework is RESTful or not as I don't want to get into a dogmatic discussion!

So far I like what I see, it is a productive framework with a nice set of features (and has cherry-picked some best ideas from other frameworks too), and what else it's Scala-based (my current favorite language).

Setup

Getting started instructions on Liftweb's wiki worked perfectly fine for me. I used instructions for Maven and I'm using Scala 2.8.

URI Matching

RESTHelper is the trait that you may want to extend for building your REST-based web services. Lift's URI dispatch follows the templating approach in which you would leave part of the URIs to be filled by the clients before they are submitted. For example, id can be sent dynamically by the user: http://example.org/api/user/{id}

The following block of code can take care of the above URI matching ...

the List of strings are the tokens after each "/" (forward slash) in the URI. The above code indicates that a GET request can handle that specific URI pattern. And when such a pattern is encountered invoke the method userDetails. If a URI is encountered as http://example.org/api/user/101, 101 is bound to id variable.

Box[LiftResponse] is the return type that's expected. Box goes by the same notion as Option in Scala. When you have stuff to return you would respond with Full(LiftResponse) and Empty when there is no response. So here is an example of how you can URI match and dispatch for a sample CRUD (I'm using simple User account management).

Ideally I would like to write this in a single case block without breaking into multiple units as I did above. When I add more than three case statements of this complexity (which is not that much, IMO) Scala compiler takes way too long and eventually gives up with an OffsetTooBigException. This issue is in bug tracker for a while now, and the above approach of splitting the matchers into multiple units is based on the workaround suggested there.

The pattern matching is flexible and works great for multiple tokens too. Something like http://example.org/user/{userId}/address/{addrId} can be extracted using a very similar pattern like above with values for userId and addrId binding to their corresponding variables.

Here is an excellent article on Scala's partial functions and pattern matching that you may find it useful in the context of this discussion.

Content Negotiation

Content negotiation is one of the core concepts of RESTful systems where a client can indicate which media type(s) it prefers. Also within the media types it can specify the order of preference. Client does this by using Accept header. For example, consider the following Accept header --

Accept: application/xml;q=0.8, application/json;q=0.9

It indicates to the server a) it can accept XML and JSON formats and b) it prefers JSON over XML (by providing higher value for 'q' parameter. q value ranges from 0.0 to 1.0, higher value indicates more preference).

Ok, so how does Lift fares in this area? Mixed results --

  • It recognizes which media type to serve by the Accept header. So for example Accept:application/json header from the client is matched to case "api" :: "user" :: id :: _ XmlJson _ => ...
  • Similarly for XML, an accept header with text/xml is matched to XmlGet method fine. One issue that I encountered here is it only recognizes text/xml as XML media type and not application/xml. application/xml is actually preferable to text/xml, generally speaking. One reason on top of my head is text/* media types ignore the encoding specified in the content, in this case if you declare a specific encoding (e.g: UTF-8) in the XML declaration header it will be ignored.
  • It doesn't respect the q parameter value. So in the above example, Accept: text/xml; q=0.8, application/json;q=0.9 it still serves the client XML as it only looks for if text/xml is present in the header.

So let's look at Tilkov's first question in the list:

Does the framework respect that an HTTP message does not only consist of a URI? I.e., is dispatch done at least based on the HTTP verb, the URI, the Content-type and Accept headers?

My answer, at this point: Yes dispatch works great in terms of -- HTTP verb, the pattern matching URI templates and Accept headers (partially). Lift has to get its act together in further tightening its content negotiation support.

[I would love to contribute some patches for this and more (like hypermedia support, will elaborate it in the future posts). Lift folks, you have a volunteer here!]

Stay tuned!

Update: Follow-up is posted here: Lift and Content Negotiation

Tags: , ,