Surya Suravarapu's Blog

Ramblings about software development and stuff related

Archive for category REST

Lift and Content Negotiation

Aug 19

Posted by Surya Suravarapu in REST, Scala | No Comments

Overview

This is a follow up of my previous post on Lift and REST discussing the aspects of URI matching and content negotiation. Lift supports Accept header of HTTP by responding back with a representation in accordance with the media type provided in the header. However, it currently doesn't support quality factor (q parameter) of the Accept header, out of the box. Here  I will attempt to provide an approach to provide that support and along the way let's explore another compelling feature of the Lift's REST support.

HTTP Accept Header

Check RFC 2616 for detailed explanation on Accept and other HTTP headers. Let's go over this based on an example: If a client sends an Accept header something like the following --

Accept: application/xml; q=0.8, application/json

is interpreted as: I prefer JSON representation but if you don't have it XML is my second choice.

Quality factor or q parameter is the one that's used to specify the preference. q is a decimal ranging from 0.0 to 1.0 and is delimited with a semi-colon (;) following the media mime type. If no q parameter is specified it defaults to value of 1.0, indicating first among the options provided.

Approach

Before going into the approach for supporting the q parameter (for your Lift-based application) let's get into one of the things that you definitely want to see in a web framework: decouple business logic from the representation. Lift doesn't disappoint you in this area. In my case I was using the same business logic, authorizing the user and making the database lookups, returning the appropriate representation independent of the business logic.

serveJx in RestHelper is there precisely for that purpose. Following code provides the URI matching rule (matches /api/user/{user_id} for GET requests) and returns an object that's of trait Convertable. Also, define an implicit def that converts the object to the appropriate representation (XML or JSON, in this case).

Relevant pieces of case class User is provided below. If you are familiar with Squeryl you may have already identified the annotations provided for the constructor arguments otherwise don't worry about it; Squeryl is an excellent Scala-based ORM (I like Squeryl quite a bit, that's a topic for another blog post!). User implements Convertable, meaning the two representations -- XML and JSON via toXml and toJson functions respectively.

So with all the above in place, the following request would result in an XML or JSON response based on the Accept header. The logic for identifying the appropriate representation  is in the RestHelper's implicit def jxSel shown below. As RestHelper does not support q parameter out of the box, UserManagementService extended from RestHelper changes the behavior by overriding jxSel.

Actual Parsing: Parsing of the Accept header and determining the representation is done using functions in the ClientMediaPreference object. Instead of embedding the code in this already lengthy post, here is a link to the gist covering the parsing logic.

Conclusion

There are a couple of areas that I still have to tighten-up the code, but that's the general idea. One of the Todo items is to send a 406 Not Acceptable response if the representation that a client asks for is not implemented by the server.

Before closing, let's continue checking off another item from Tilkov's litmus test (as we did in the last post) ...

Can I easily use the same business logic while returning different content types in the response?

Answer: Yes, the framework is flexible in this aspect.

Tags: , ,

Lift and REST: URI Matching and Content Negotiation

Aug 3

Posted by Surya Suravarapu in REST, Scala | 9 Comments

As a Scala enthusiast I'm currently evaluating Lift, particularly the aspect of its REST support. This could become a series of posts on the topic as I try to understand the framework better and may be subject it to a litmus test proposed by Stefan Tilkov. Of course, I will not be making a judgment call whether the framework is RESTful or not as I don't want to get into a dogmatic discussion!

So far I like what I see, it is a productive framework with a nice set of features (and has cherry-picked some best ideas from other frameworks too), and what else it's Scala-based (my current favorite language).

Setup

Getting started instructions on Liftweb's wiki worked perfectly fine for me. I used instructions for Maven and I'm using Scala 2.8.

URI Matching

RESTHelper is the trait that you may want to extend for building your REST-based web services. Lift's URI dispatch follows the templating approach in which you would leave part of the URIs to be filled by the clients before they are submitted. For example, id can be sent dynamically by the user: http://example.org/api/user/{id}

The following block of code can take care of the above URI matching ...

the List of strings are the tokens after each "/" (forward slash) in the URI. The above code indicates that a GET request can handle that specific URI pattern. And when such a pattern is encountered invoke the method userDetails. If a URI is encountered as http://example.org/api/user/101, 101 is bound to id variable.

Box[LiftResponse] is the return type that's expected. Box goes by the same notion as Option in Scala. When you have stuff to return you would respond with Full(LiftResponse) and Empty when there is no response. So here is an example of how you can URI match and dispatch for a sample CRUD (I'm using simple User account management).

Ideally I would like to write this in a single case block without breaking into multiple units as I did above. When I add more than three case statements of this complexity (which is not that much, IMO) Scala compiler takes way too long and eventually gives up with an OffsetTooBigException. This issue is in bug tracker for a while now, and the above approach of splitting the matchers into multiple units is based on the workaround suggested there.

The pattern matching is flexible and works great for multiple tokens too. Something like http://example.org/user/{userId}/address/{addrId} can be extracted using a very similar pattern like above with values for userId and addrId binding to their corresponding variables.

Here is an excellent article on Scala's partial functions and pattern matching that you may find it useful in the context of this discussion.

Content Negotiation

Content negotiation is one of the core concepts of RESTful systems where a client can indicate which media type(s) it prefers. Also within the media types it can specify the order of preference. Client does this by using Accept header. For example, consider the following Accept header --

Accept: application/xml;q=0.8, application/json;q=0.9

It indicates to the server a) it can accept XML and JSON formats and b) it prefers JSON over XML (by providing higher value for 'q' parameter. q value ranges from 0.0 to 1.0, higher value indicates more preference).

Ok, so how does Lift fares in this area? Mixed results --

  • It recognizes which media type to serve by the Accept header. So for example Accept:application/json header from the client is matched to case "api" :: "user" :: id :: _ XmlJson _ => ...
  • Similarly for XML, an accept header with text/xml is matched to XmlGet method fine. One issue that I encountered here is it only recognizes text/xml as XML media type and not application/xml. application/xml is actually preferable to text/xml, generally speaking. One reason on top of my head is text/* media types ignore the encoding specified in the content, in this case if you declare a specific encoding (e.g: UTF-8) in the XML declaration header it will be ignored.
  • It doesn't respect the q parameter value. So in the above example, Accept: text/xml; q=0.8, application/json;q=0.9 it still serves the client XML as it only looks for if text/xml is present in the header.

So let's look at Tilkov's first question in the list:

Does the framework respect that an HTTP message does not only consist of a URI? I.e., is dispatch done at least based on the HTTP verb, the URI, the Content-type and Accept headers?

My answer, at this point: Yes dispatch works great in terms of -- HTTP verb, the pattern matching URI templates and Accept headers (partially). Lift has to get its act together in further tightening its content negotiation support.

[I would love to contribute some patches for this and more (like hypermedia support, will elaborate it in the future posts). Lift folks, you have a volunteer here!]

Stay tuned!

Update: Follow-up is posted here: Lift and Content Negotiation

Tags: , ,

REST: DELETE operation and tunneling

Oct 22

Posted by Surya Suravarapu in REST | 6 Comments

I was looking at some presentation slides on REST vs SOAP and one of the major drawbacks listed for REST approach is -- lack of ability to use a message body for DELETE operations. I was not sure when I read that, why that would be a drawback.  (Discussion on the listed drawbacks will be a separate post by itself for another day).

A DELETE operation might look something like the following, where 123 is the ID of the customer:

DELETE /customers/123
Host: example.com

I was thinking about some use cases where a server might need more information for deletion. Thinking along the lines I tweeted this primarily to contest the presenter's belief that it is such a huge drawback that you use that as a strike against REST-based approach.

Subbu Allamaraju responded. He says that the the question is a valid one.  Subbu said that:

That is a good question. Think of any case where client has to explain why the resource needs to be DELETEd. This is not uncommon.

Assuming that's a valid concern I was thinking of ways to do that in a RESTy manner. Only way that I could think of at that point was whether we could use POST and perform DELETE. That sounded to me like tunneling. Tunneling is hiding operations from HTTP. There is no way to know whether the operation is -- safe, idempotent, both safe and idempotent, neither safe nor idempotent.

As we continue discussing this on Twitter I've asked how different this scenario would be from tunneling via GET, something like the following:

GET /customer?method=delete&id=123
Host: example.com

The above GET is a clear example of tunneling. Similarly, SOAP-way of POST is another good example of tunneling.

POST /CustomerService HTTP/1.1
Host: example.com
Content-Type: application/soap+xml; charset=utf-8
Content-Length: nnn

<?xml version="1.0"?>
<soap:Envelope
 xmlns:soap="http://www.w3.org/2001/12/soap-envelope"
 soap:encodingStyle="http://www.w3.org/2001/12/soap-encoding">
    <soap:Body xmlns:m="http://example.com/customer">
        <m:deleteCustomer>
            <m:id>123</m:id>
        </m:deleteCustomer>
    </soap:Body>
</soap:Envelope>

Approach

An approach that sounded reasonable: use POST with a distinct URI when in doubt. That way you would avoid tunneling by making it a distinct resource. 

POST /customers/123/deleteme
Host: example.com
Content-type: xxx

[send reasoning to the server why the resource is being deleted in the body]

This provides some visibility into the operation, via a URI that conveys the intention.

One downside of this approach is caches will not see the resource being deleted. In spite of that, this approach seems reasonable when you have a specific need to address the use case in question.

Thanks to Subbu for suggesting this approach. Provide your comments if you know of any other approaches.

P.S: Just one more reason why I like Twitter! Feel free to follow me there.

Tags:

Java’s HTTP Handler and Cache Validation Issues

Oct 8

Posted by Surya Suravarapu in Caching, Java, REST | 6 Comments

Background

A little while ago I've mentioned that I was working on client-side HTTP caching (using Ehcache) for REST clients. After a little hiatus, I'm back to complete the unfinished business, precisely dealing with cache validation support (using ETag, Last-Modified, If-None-Match, If-Modified-Since headers). I've also explained the approach I was taking to implement the solution, using Java's ResponseCache mechanism.

However, I think I've hit a dead end implementing the solution using that approach. I will try to explain it here and hope that smarter people out there provide their thoughts.

Overview

Let's start with a simple straightforward scenario. Quick control flow of the Java protocol handler approach:

  1. A client application gets an instance of sun.net.www.protocol.http.HttpUrlConnection, which extends java.net.HttpUrlConnection, via url.openConnection(). This hanlder instantiates a registered instance of java.net.ResponseCache, if there is one available.
  2. When a request is sent to the server via HttpUrlConnection, protocol handler first checks whether the representation is present in the cache by calling the get() method of the ResponseCache. If it is in the cache send it to the client, else send the request to the origin server.
  3. If the request is sent to the origin server, and if the response is any of 200, 301, or other "cache-able" statuses, the handler then calls the put() method of the ResponseCache to potentially cache the representation.
  4. ResponseCache would store the element in the cache. It uses Expires, Date, Cache-Control headers to determine time to live and set it on the element. Let's ignore expiration model for this post as the focus is on validation.

Note: You have to write your own concrete implementation for ResponseCache to store and retrieve elements from the cache. Java doesn't provide an out-of-the-box implementation for it, but it provides a framework for doing so.

Validating Cached Element

Now let’s look at a scenario that does cache validation. First, what is validation? There are two headers a server may send for validating the resource: a timestamp (Last-Modified) indicating when the resource was last changed, and an entity tag value (ETag). Server may choose to send only one of these headers, as both of them try to achieve the same purpose.

Responding to a request for resource X, a server sends along one or both these headers to the client along with X’s representation. On any subsequent request for resource X -- the client may honor these response headers, and sends two of its own headers: If-Modified-Since (with the value of the Last-Modified header) and If-None-Match (with the value of ETag header). Former requests the server to send the representation only if the resource is modified since the Last-Modified time it has got, and the latter asks to send the representation only on the change of ETag value that it supplied.

If there is a change in the resource, a server sends an updated representation, with new values for ETag and/or Last-Modified headers. This scenario works fine with no issues as you get a 200 response back, and the protocol handler handles this just fine (similar to the straightforward scenario mentioned above). The issue that I'm going to mention is with the case in which the server determines that there is no change with the resource, and sends back a status 304, NOT MODIFIED, with no body in the content.

See the following sequence of events that end up with a status code 304 from the server (click on the image to enlarge):

Conditional GET

Issues with Java's HTTP Handler

  • A client or client-side cache should first check whether a cached representation is available before sending a conditional GET of this sort. (There is no point sending Not-Modified-Since and/or If-None-Match headers if it doesn’t have a representation to fall back on). Java’s cache handler framework using HttpUrlConnection does not provide an option to do so.Let's see the relevant source code of sun.net.www.protocol.http.HttpUrlConnection, lines 399-410: 
    // Set modified since if necessary
long modTime = getIfModifiedSince();
if (modTime != 0) {
    Date date = new Date(modTime);
    //use the preferred date format according to RFC 2068(HTTP1.1),
    // RFC 822 and RFC 1123
    SimpleDateFormat fo = new SimpleDateFormat(
        "EEE, dd MMM yyyy HH:mm:ss 'GMT'", Locale.US);
    fo.setTimeZone(TimeZone.getTimeZone("GMT"));
    requests.setIfNotSet("If-Modified-Since", fo
        .format(date));
}

    The above block of code adds If-Modified-Since header but makes no checks whatsoever whether the representation is available in the cache.

  • I don't see a reference to If-None-Match header in the source. So if the client sends that header, it will be sent to the origin server without an availability check
  • In case, if there is no representation in the cache, the cache must have an ability to remove the validation headers from the request before sending the request to the origin server. I don't see this framework supporting such a behavior.

Thoughts??

// Set modified since if necessary
0400:                    long modTime = getIfModifiedSince();
0401:                    if (modTime != 0) {
0402:                        Date date = new Date(modTime);
0403:                        //use the preferred date format according to RFC 2068(HTTP1.1),
0404:                        // RFC 822 and RFC 1123
0405:                        SimpleDateFormat fo = new SimpleDateFormat(
0406:                                "EEE, dd MMM yyyy HH:mm:ss 'GMT'", Locale.US);
0407:                        fo.setTimeZone(TimeZone.getTimeZone("GMT"));
0408:                        requests.setIfNotSet("If-Modified-Since", fo
0409:                                .format(date));
0410:                    }

Tags: , ,

REST: CRUD with Grails

Jun 8

Posted by Surya Suravarapu in Grails, REST | 6 Comments

Several weeks ago I have posted my experiences with JAX-RS. Later on, as I read and understood more about Grails, I wanted to perform a similar CRUD exercise with Grails. Grails supports REST out of the box. If somebody says that Grails rocks, you don't have to take them at their word but try out a few examples like this (and eventually agree with them!).

Obviously, REST is lot more than CRUD, and an excellent architectural pattern in my opinion. But for this post we will restrict ourselves to the CRUD aspect.

Get started

Downloading and setting up Grails is out of scope for this post. Refer to Grails website for that. Once you have Grails setup run

grails create-app [app-name]

Replace [app-name] with what ever you want to name your app. This command should create a standard Grails project structure. We will use the embedded database (HSQLDB) and embedded servlet container (Jetty). Grails ships with them auto-configured. So use all defaults for this exercise.

Resource

The resource we will be dealing here is a simple Customer object with firstName, lastName and zipcode properties. Declare Customer.groovy in grails-app/domain. 

class Customer {
  String firstName
  String lastName
  String zipcode

  static constraints = {
    firstName(blank: false)
    lastName(blank: false)
    zipcode(blank: false)
  }
}

All the properties are made mandatory, as specified in the constraints above.

URL Mappings

URL mappings in Grails is an extremely useful and powerful feature. We can set a URL mapping's action parameter to a map that associates HTTP methods with action names. UrlMappings.groovy will go into grails-app/conf 

class UrlMappings {
    def mappings = {
      "/customer" (controller: "customer") {
        action = [GET:"list", POST: "create"]
      }

      "/customer/$id" (controller: "customer") {
        action = [GET: "show", PUT:"update", DELETE:"delete"]
      }
	}
}

Here we have mapped /customer to a corresponding controller (we still have to work on the Controller, coming next), and the actions are mapped such that GET requests go to list action of the controller and POST is forwarded to save action. For a different URL pattern /customer/$id we have also provided mappings for its supported HTTP methods: GET, PUT and DELETE.

Controller

Create CustomerController.groovy in grails-app/controller, and let's start working on CRUD operations on our resource.

Create

import grails.converters.XML

class CustomerController {
  def create = {
    def xml = request.XML
    def customer = new Customer()
    customer.firstName = xml.firstName.text()
    customer.lastName = xml.lastName.text()
    customer.zipcode = xml.zipcode.text()

    if (customer.validate()) {
      customer.save();
      response.status = 201
      render customer as XML
    } else {
      sendValidationFailedResponse(customer, 403)
    }
  }

  private def sendValidationFailedResponse(customer, status) {
    response.status = status
    render contentType: "application/xml", {
      errors {
        customer?.errors?.fieldErrors?.each {err ->
          field(err.field)
          message(g.message(error: err))
        }
      }
    }
  }
}
  • See the power of Groovy, the ease with which you can navigate XML. Populate the Customer object and validate it for constraints defined in the domain object.
  • If the validation is passed save the resource, set the status and respond back with the created resource (with id). In this context, see line 14. It can't get any simpler than that. A domain object is transformed to XML with just that call.
  • Grails ships with two converters XML and JSON. If you want JSON response above just replace line 14 with render customer as JSON
  • rest of the code is status setting and error handling, a display of Groovy closures.

Test it
First, run your app from the command line --

grails run-app

Rest-client is an excellent REST client, I strongly recommend it for standalone testing. It has got a nice little UI and works as advertised with little fuss.

Create Customer Resource

(Click on the image to enlarge)

Read

  def list = {render Customer.list() as XML}

  def show = {
    Customer customer = Customer.get(params.id)
    if (customer) {
      render customer as XML
    } else {
      SendNotFoundResponse()
    }
  }

  private def SendNotFoundResponse() {
    response.status = 404
    render contentType: "application/xml", {
      errors {
        message("Customer not found with id: " + params.id)
      }
    }
  }
  • It doesn't get much simpler than that. Two actions list and show display XML but act on two different URL patterns (see URL mappings). Former lists all the customers in the database and the latter provide details of a specific customer (with a given id).
  • Note that id portion is passed on using params.id field
  • Status code 404 is sent back, see SendNotFoundResponse. Response is sent back in XML.

Test it
GET is easy to test in a browser, or use Rest-client to test. See the URL, client is requesting for a resource with an Id.

Get Customer Resource

(Click on the image to enlarge)

Update
Update and Delete are very similar to the code described above. The following should be self explanatory by now.

  def update = {
    Customer customer = Customer.get(params.id)
    if (!customer) {
      SendNotFoundResponse()
    }

    def xml = request.XML
    customer.firstName = xml.firstName.text()
    customer.lastName = xml.lastName.text()
    customer.zipcode = xml.zipcode.text()

    if (customer.validate()) {
      customer.save();
      response.status = 201
      render ""
    } else {
      sendValidationFailedResponse(customer, 403)
    }
  }

Delete

  def delete = {
    Customer customer = Customer.get(params.id)
    if (!customer) {
      SendNotFoundResponse()
    }
    customer.delete();
    response.status = 204
    render ""
  }

Tags: ,

RESTful API and HATEOAS Conundrum

Mar 25

Posted by Surya Suravarapu in REST, WebServices | 2 Comments

HATEOAS --- Hypermedia as the Engine of Application State is an important constraint of REST architecture as defined by Roy Fielding. To better understand the principle I've been discussing the topic with various REST enthusiasts. In this post I would like to provide an update on some of the learning points. There are arguably quite a few false claims of RESTful-ness (by API providers) while not following some of the basic tenets of REST. HATEOAS seems to be one of the main principles that is either misunderstood or not implemented in its true spirit.

Hypermedia and Application State

There are two important pieces here:

  • What is Hypermedia?:  Just as Hypertext documents contains links to other chunks of text, Hypermedia links to many media types including -- text, images, audio, video, etc. WWW is a common example of Hypermedia.
  • What is application state?: Not to be mistaken with the resource state. Application state is the state of the user's application of computing to a given task. This is an important distinction, as Mark Baker explains in this InfoQ article:

"application state" refers to the state that determines "where" the user is in the process of completing a task. For example, when doing personal banking, is the user currently viewing account balances, filling in a bill payment form, or about to order new cheques? Those are each different application states. Some people mistakenly believe that "state" here refers to resource state, which would include, in this example, the balance of the accounts or the list of recent payments made. That isn't the case.

HATEOAS achieves two very important aspects --- protocol is not stateful, and loose coupling of client-server can be achieved as the server evolves describing new URIs.

Source of some confusion

Roy Fielding published a post in an effort to clarify what a true RESTful API is and how people are abusing the term. The last bullet point of the post says that ..

A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). ...

That along with some related discussions raised some interesting questions in my mind about how much prior knowledge that a client can have.

Ask an Expert

I have asked Subbu Allamaraju a couple of questions in this regard (Subbu is well known in this area, I learnt a lot from his blog posts and articles. He works at Yahoo developing standards, patterns and practices for HTTP web languages, and is extremely passionate about REST).

Question: How much information should the client know? If I strictly go by Fielding, it should only know the start page, and server should take care of the rest, driving the flow?

Subbu: That may be possible, provided the client knows all the link relations and media types. But there is no such constraint as having one staring URI. The idea is to keep representations contextual, but not necessarily to limit the number of prepublished URIs to one. When a representation is contextual, the client can decouple itself from server's business rules.

In other words, what a client can do on a given resource depends on some business rules on the server. Instead of the client guessing that it can do certain action on a given resource, it can rely on the links contained in the representation. That is, by embedding application state and context into representations, servers can avoid leaking business rules to clients. That is the main benefit of HATEOAS for client-server apps.

[Next question could sound a bit extreme to some folks, but I read the idea earlier and so wanted find Subbu's take on that.]

Question: Is it practical to say that the client shouldn't know anything including the fields that are being worked on (say username and password fields of the form) and shouldn't know about what page it is posting to, say "/login".

Subbu: This may be not practical and can be prohibitively expensive.

Media types, link relations and things like microformats can help to a certain extent. In the case of XHTML representations containing well-known microformats, you may be able to minimize the amount of information a client will need to know to intelligently interact with a RESTful server. Even on the web, given a perfectly described XHTML, try replacing a human with a perl program. How far can that perl script go? Can it pass the Turing test?

If automation and self-discovery is the goal, of course, it may be worth trying. But I doubt if every application needs that capability.

[Subbu's Describing RESTful applications is an excellent article that explains HATEOAS and more. Highly recommended]

Next Steps

Along the lines I will be closely following Solomon Duskis' thought process to see what his experiments result in. Solomon and I had some good conversations, and his posts helped provoke some thought.

I guess the best way to understand this is by reviewing a good example. I've asked the following question on Twitter

If asked about an example of truly RESTful API, which (publicly available) API do you point to? Interested: no-state-on-server, HATEOAS.

.. for which I got a couple of responses --- Amazon S3 and NetFlix. Did some research and found some good reviews on NetFlix API, so will explore that further ..

Tags: ,