Although I found all the sessions I attended very inspiring, my personal favourite has to be the debate on “democratic identity” chaired by @curiousc. I found this debate intellectually stimulating because identity management is something I have been interested in from a technical perspective for some time. My focus has been focussed using identity management techniques to help make web applications convenient for users to use, with the belief that this will increase convergence. However, I had never approached it from a democratic perspective before.

The basic premise was whether or not an online posting made anonymously should bear any weight in democratic debates. There were different opinions expressed, but all centred around the following main points.

• The content of the posting or quality/reputation of the online community in which the posting was made is of more significance than knowing the individual’s identity
• The identity of the individual can be vouched for by an external identity provider service in accordance with the community’s identity requirements

Although the first point would be interesting to debate more, the second point is more applicable to my personal interest in identity management.

The general premise is that the individual should be able to post anonymously within the community. For some, anonymous postings may be the only way their voice can be heard without putting themselves in danger. What’s important to realise is that this does not mean that the individual will be posting without any form of handle. They might be called “fred_flintstone”, but the discussion point made can be just as genuine as that of someone using their real name. In fact, the individual may not think the discussion point is of any real significance at the time of posting it, but then it turns out others have had similar thoughts and the points are explored further, increasing the profile of the debate.

If the profile of the debate increases, someone might challenge the validity of the individual’s participation in the debate.  For example, if the debate is about disturbances in a particular neighbourhood, others might challenge the individual on the basis that they believe s/he does not live in the neighbourhood and hence their description of events is unlikely to be accurate. In such a situation, an external identity service may be used by the individual to link their “anonymous” handle with a set of identity assertions such as “individual lives within X miles of point Y” where Y represents the centre point of the neighbourhood. As long as there exists trust between the community and the external service provider, this assertion should be a enough for the community to take the individual’s description of events seriously whilst the individual’s identity is kept protected.

I wonder if there exists a social space where such an approach has been implemented?

The focus was on strengthening the community of developers and designers which already exists in the South West.  There were a series of 3 minute “lightning talk introductions” from individuals and small enterprises from the region, followed by some more lengthy spotlight presentations and workshops.  The conference wrapped up with some 2 minute pitches by representatives of other web related communities of potential interest to attendees.

I would highly recommend that if you live in the South West region that you subscribe for the next #digpen unconference which is on Saturday the 11th of June in Cornwall.  For up to date information please follow the hyperlink in the reference section.

I attended this event to gain more awareness of the types of technologies and tools that freelance and SME web developers are using to build and deploy web apps for clients.  I am especially interested in this as I would like to help make possible a future where local authorities play a key role in facilitating growth of local communities such as #digpen – procuring digital services from the community to help deliver innovative and effective services to the public.

Knowing what technologies freelance developers and SMEs currently favour enables myself and my team to strategise and build the case for a platform on which these developers can deploy their web app solutions without them needing to invest in niche technologies or bespoke integration protocols.  i.e. moving to a point where local authorities provide a thin layer of open standards driven technology on top of which developers can make use of their existing preferred frameworks and technologies to deliver web apps.

What is in it for the local authority?

Local authorise provide a vast number of services to the public.  These are a mix of services that the authority are required to provide (legislated by central government) and services driven by local initiatives.  Since the services are diverse, delivering digital services effectively means investing in solutions architecture – for internal developments.  This helps reduce the cost of IT support and improves reliability and agility in meeting ever changing business requirements.  An example of this is the UK e-Government project which took place 2004/2005 and introduced a technical architecture which enabled 140+ transactional services to operate with a shared codebase.

So within local authorities, there has existed effective solutions architecture for quite some time.  Unfortunately it’s often the case that this level of standardisation is not found in the infrastructure that is hosting procured solutions.   The reasons for this are many, but in my opinion, the most significant being that technical fit against existing technical architecture & strategies is considered lower importance than meeting specific business requirements at that time – they should be the same thing!

So we have a situation where more and more stress is put upon internal IT resources to keep the infrastructure on which these procured solutions are hosted ticking over.  This in turn means that internal IT cannot be agile and provide a coherent service to the business.

To counter this problem, local authorities must communicate to potential web app suppliers an explicit and finite set of parameters which they must operate within.  A service delivery platform enables this to take place.  The challenge is to architect the platform such that it meets the objectives explained above whilst not simply moving the burden to the supplier.  i.e. desensitising the supplier from providing the solution to the local authority due to increased development/support costs or inability to resell their solution.

To summarise – the benefits include reduced cost and increased agility.

What is in it for the supplier?

The cabinet office ICT strategy 2011 dictates that there should be a preference for smaller projects over larger projects and empowerment of SMEs to deliver competitive solutions.  So the main beneficiary of the suggested platform will be the SMEs and freelance developers.

A real issue with UK procurement is that a large proportion of procurement is done via tenders.  The tender framework was designed to make the procurement exercise fairer – unfortunately it has made it more onerous for SMEs to compete for business.  Producing a tender response entails a substantial amount of effort on behalf of the SME who often do not have the infrastructural investment to do so easily.  Producing a response does not guarantee work so has to be weighed against the cost of producing the response.

So how can we cut through the red tape and get SMEs more involved in web app solutions delivery?

I would propose the best solution is to provide easy access to the aforementioned platform.

This way the SME gains an awareness of the kind of technical solution the local authority desires.  This in turn enables them to focus their effort on matching their capabilities against the specific business requirements described by the tender.  The result being that a more educated decision can then be made by the SME on whether or not to commit resources to completing a tender response.

It is fully appreciated that many SMEs will already have technology and framework preferences (why I attended the #digpen conference), and like any supplier, will want to be able to resell their solution to more than one client.  This is why the platform must be very light weight and offer a large number bridges to all the popular frameworks.

So modifying an existing web app development to be compliant with the proposed platform will not require much, if any, effort.

For SMEs who are developing web apps on a per client basis (or want to provide extra value to their clients by integration with existing solutions in use at the authority), the proposed platform will provide a lot of readily accessible services to the SME developers, helping to cut down the amount of development time needed.  For example authentication & authorisation services and access to open data and functionality from other deployed solutions.

Finally, by delivering a solution which is in line with the local authority’s internal technical architecture, this reduces (potentially removes) the need for authority’s IT staff to be involved in changes to the solution.  This enables the supplier to be agile in meeting new business requirements that emerge over time – helping maintain good ongoing relations between the supplier and local authority.

To summarise – the benefits include enabling a more educated decision on whether or not to compete in tenders, faster development through technical support services and improved ongoing relations between the supplier and the local authority.

On to discussions!

I believe that this article has only scratched the surface of what benefits can be realised if local authorities standardised on a well designed platform for service delivery.  I’d be very interested to hear your views on the matter…

Reference

If you think you might be interested in attending the next #digpen unconference check out the following links:

• Digpen III information on Eventbrite
• Digpen II video coverage on YouTube

If you work as a technical lead or a motivated developer for a local authority and would like to help make the platform a reality then join the IT Solutions Architecture Community of Practice

Service providers commit a lot of resources into keeping their customer data up to date, so that it can feed into delivering a more streamlined service to the individual, facilitate BI and power campaigns. There appears to now be a realisation that this approach is both expensive and, in reality, ineffective. They’re always playing catch up.

The ownership and management of this information shouldn’t be with the service provider. The information belongs to the individual, and how and when the information is used should be managed by the individual.

Are we on the brink of a personal information management revolution?  The team at Mydex, a CIC (Community Interest Company), certainly present a good case, conceptualised as a PDS (Personal Data Store).

Reference

If you are interested in reading more about the concept of Personal Data Store, follow the links below:

• MyDex’s official website
• “The case for Personal Information Empowerment: The rise of personal data store”

I had never attended a mashup event before so this seemed like a really good opportunity, right on my doorstep!

Experiencing the event…

The Data Mashup was a great success overall, but I cannot take much credit for that.  Other commitments meant that I could only assist the team for the Sunday.  Despite this, I found the event exhilarating, and a real insight into both the local talent and enthusiasm that exists for events such as these.

The event was very well organised by the team at Forward 25, and likely to be one of many.  Before joining the team, my expectations were that I would be the odd one out and others would have had been there for the duration.  This however wasn’t the case at all.  There was a core team who had dedicated their time to the event, but many others had come for only a day or two.

I wasn’t expecting to get involved in the development so late in the event.  The more dedicated developers at the event would already be battling against the clock to deliver functional apps for the following day – and likely would not appreciate needing to take time out to bring me up to speed.  I would have preferred to add value by to the event as a tester benefitting from the fact that I had no prior knowledge of the developments.  Unfortunately the development had not advanced to this stage yet.  Completing an end to end app in just over 3 days was always going to be challenging.

In the end I thought I would attempt to setup a JSR-286 portlet development environment to offer an additional delivery approach for the midwifery data.  Unfortunately this was a futile attempt because installing Liferay (preferred Portal server), Eclipse IDE, Liferay IDE plugin for Eclipse, and carry out the actual portlet development all in one afternoon was simply not possible.  The attempt did however attract some interest from the fellow Java developers at the event who were new to portlet development; so I feel the effort was not wasted.

Can the event be improved?

As mentioned earlier, the event was organised very well by Forward 25, but as a first of its kind in Exeter, it would seem sensible to reflect on the experience with a view to hopefully making the next mashup event will be even better!

So here are some of my observations / suggestions:

• It would be useful if some preliminary information about the dataset feeding into the mashup was made available before the event.  Maybe as a separate presentation & registration event.  This would also provide an opportunity to identify which technical skills can be called upon before the actual event starts.

• There should be an emphasis on documenting the high level design of the app(s) at the end of the design phase.  This document could be considered an induction document for developers who arrive later in the event.

• As this is a data mashup, it would be useful if the raw data is readily available via APIs or easily accessible databases; rather than the identification & setup of suitable services taking part during the event and deducting from the time available for the development of the apps.

Reference

If you’re interested in reading more about this mashup event, check out the following links:

• Forward 25′s official website
• “App Development for Upcoming State of the World’s Midwifery Report” – A report by Demotix

I would like to propose that modern portal and portlet technology standards as key to a realistic solution to the challenging problem of achieving a “single view” of the customer. The technique I will describe can be extended to any type of object whose information is spread across disparate applications of which the solutions architect has often very limited control.

There has been substantial discussion on this topic, but in my researching experience, few approaches have emerged that propose a solution to “single view” when systems have not been specifically architected to fit into such a solution.

This is especially important for local government organisations, where there exists a substantial number of 3^rd party supplied applications; often caused by a lack of corporate governance on procurement of applications.

Often, in order to make information held in such applications searchable, the appropriate information is extracted from the application and indexed externally. This is identical to how popular www search engines such as Google provide search capability. However, this search strategy is not fit for organisations that take access security seriously.

Security can quite simply be split into two categories: Infrastructure and application security. Infrastructure security prevents unauthorised access to environments in which information resides in data stores. This includes preventing direct access to enterprise data stores or control of servers. If your infrastructural environment is not secure, then it is irrelevant what degree of security you apply at the application level. However, it is out of scope of this discussion.

Application security is however far more interesting in my opinion. Applications dictate the security model through which information access is facilitated. They are the gatekeepers. It is important to consider that whenever information is retrieved from such an application, it leaves its secure environment. For this precise reason, the usage of centralised indexing technology must be considered a security risk to sensitive enterprise information.

So how can we balance security with usefulness? Bring on federated systems!

Federated Search Strategy

Before discussing what value a federated search strategy can bring to an enterprise, it is important to understand what the term “federation” means.

“[A federation is] the formation of a political unity, with a central government, by a number of separate states, each of which retains control of its own internal affairs.” (sourced from www.dictionary.com)

This definition is highly applicable in the context of solutions architecture; where there is a need to facilitate information and process governance.

Interestingly, as mentioned earlier, information governance (of sort) is often forced upon the systems architect in the form of unmodifiable 3^rd party supplied applications promoting their own data stores for information. Unfortunately it is not useful information governance, as it leads to information redundancy which causes issues when providing a comprehensive view of information held on a subject or customer.

However, this form of governance is useful in the area of security. The application, although potentially duplicating information held elsewhere, has been carefully designed (if it holds sensitive information) to prevent unauthorised access to the information it holds. Consequently the application would naturally define user types or roles which govern user permissions. I will hereafter refer to these as “application security roles”.

Federated search is all about allowing each application to govern its own search functionality, on its own data, according to its own rules.

So how does federated search relate to the single view of the customer you could ask? In generic terms, a customer is just another object type modelled by the organisation’s applications.

An organisation has its own roles that make sense at the organisational level. These could be described as “organisational security roles”. These are rarely the same as the application roles, so a mapping scheme between the two sets is required.

Commonly, in a federated search system, a user enters a search criteria once into a common component such as a form field in the header of a website. The system then delegates the search onto all the participating applications. In the event of the user having been authenticated previously, this delegation will include specific application roles identifiers that are derived through the mapping scheme discussed earlier. If no such authentication took place, then a “guest” role is used. The system then waits for all the results to become available before displaying a consolidated results list back to the user.

One challenge is dictating what fields are acceptable search criteria and provide appropriate form field(s) for the user to express their search criteria through. If you are searching for information on a particular customer, then it would make sense to provide search fields for information such as: first name, last name, and of course, customer reference.

Yes that would be very nice – but entirely unrealistic!

The reason being that not all applications participating in the federation system will store a customer reference against a customer object. It’s likely to store a reference of some sort, but there are no guarantees that it will even be in the same format as that of its counter parts from other applications in the federation. In addition, customer’s ID within an application are often generated rather than input by the administrator of that application.

In my opinion, the only way around this issue, is an index of cross references needs to be maintained externally to each application. This is unfortunate, because as mentioned earlier, this information then bypasses the application’s security model. Only the IDs should be kept in such an index to mitigate this risk.

On the flip side, one could consider such an index to be an application in its own right, which participates in a federation. After all, the cross referencing is in itself information, and access to it should be governed by user security roles.

Another issue is that your search criteria fields must refer to data items common to all the applications in order to ensure searches are comprehensive. Otherwise, only a subset of applications will retrieve information on the customer. Again there is a way to work around this, but it involves storing meta information about what fields exist in what applications and then doing preparatory searches against at least one of those applications to return an ID which is common to all the federated applications (potentially requiring further cross referencing indexes) and which can be used in the main search. In addition this approach would need to define precedence rules for which systems to carry out the preparatory search in when more than one option is available. In other words, you can very easily end up with a very complex implementation which grows exponentially in complexity whenever new searchable fields need to be added.

There must be a better way…. and there is!

Federated search through modern portal technology

In recent years, portal standards have matured substantially with the advent of web 2.0 technologies and increased user expectation for seamless integration between applications.

Portals are often seen as simple content aggregation engines. However, they are capable of much more. With the release of the JSR-286 (a.k.a. portlet 2.0) standard in June 2008, portals became capable of acting as messaging brokers to enable IPC (Inter-Portlet Communication) without reverting to proprietary implementation as was the case with JSR-168 (a.k.a. portlet 1.0).

It is common practice to make use of messaging technologies in middleware to benefit from

• reliable delivery of information
• asynchronous processing
• loose coupling of component parts/services

With the advent of JSR-286, we are starting to see the same benefits in the development of presentational components that can be mashed up to produce useful composite applications.

The means through which this is achieved is JSR-286 eventing for IPC. Portlets are now able to trigger named events in the scope of a namespace. These events can transport any object as long as it implements the Serializable interface.

I would suggest that Apache XMLBeans can provide ideal object types to send as payload. It fully implement the XML Schema (XSD) specification and hence can be used to model very complex information structures.

So how do namespaces relate to the concept of federated search? In a federation of applications, namespaces are going to be vitally important due to the high likelihood of identically named fields between systems. Namespaces give context. Two types of contexts should be appreciated:

• Organisational context
• Application context

It is the role of the solutions architect to define an appropriately designed organisational context namespace in which all the object types of interest to the organisation can be represented.

Likewise, it is the role of the application developer or supplier to define an appropriately designed application context namespace in which all the object types of interest to the application can be represented.

In order to facilitate effective loosely coupled IPC, applications should only ever have their namespace translated to the organisational namespace and never directly to the namespace of another application. At this time, I believe this is best achieved through broker portlets that must be placed on the same page (JSR-286 event messaging is page scoped).

Earlier in this article I use the example of customer references not matching up between applications. A solution to this would be to deploy a broker portlet whose job is to listen for events in a specific application’s namespace and translate these into the organisational namespace and finally to the namespaces of other applications. Once translated, the portlet can create events that are indifferent to events originating from the destination application’s own portlets.

In fact, if federated search is implemented through JSR-286, then there is no real need in passing anything but cross references between portlets. The less information you pass the better from a security point of view.

If only references are communicated, then the application’s portlet(s) will be needed in order to display any information from its secure data store relevant to the search. This means that the “application security roles” assigned to the user will dictate how much or how little information is shown. In addition, you get instance gratification should the user’s “application security roles” change in the middle of a search.

The final piece of the puzzle is how to initiate a search in the first place. The traditional concept of a single search box bears little relevance in a JSR-286 federated search. I would argue that it should be possible to initiate a search from any application’s portlet that you have access to and that once an object of interest has been selected within that portlet, the eventing described earlier should take place to propagate that selection across all portlets on the page being viewed. With such a design, it is possible to drill down through the information in a recursive fashion. Much like how one browses the world wide web.

Maybe the concepts discussed in this post could be described better as federated browsing triggered by an initial application centric search?

On to discussions!

I hope you have enjoyed reading my thoughts on this matter. I think it’s an interesting topic, and I’m very aware that I’ve only just begun to scrape the surface of it.

I’m currently working on architecting a new web delivery platform built upon open standards and technologies including XHTML, AJAX, JSR-286 (Portlet 2.0), WSRP, SOAP, REST, xForms, MVC.

This new web architecture must not only enable web developers to deliver reliable applications through the web channel, but extend to empower non-technical users to build mash-ups through configuration of common solution web components.

Stay tuned!