Viewing entries tagged
developer

2 Comments

Library Oriented Architecture

Library Oriented Architecture Icon

Library Oriented Architecture may sound like yet another buzzword in the software arena, but one that is not properly documented as of yet. It is not a common term and certainly far from the widely popular SOA or Service Oriented Architecture. Since there is no formal definition on the term LOA, I’m going to take a stab at it:

“Library Oriented Architecture defines the methodology for creating software components in the form of reusable libraries exclusively constrained to a specific domain ontology.”

What does it mean? Well, the part about ontology I’m not going to drill too deeply into that, in a nutshell “don’t confuse a contact with a user, they belong to different domain ontologies” (I wrote a different article about it HERE). In this piece we are going to drill down into the software piece, the separation of concerns, and how to define a practical framework to create things in the right places.

I caught the term LOA for the first time at SuperConf 2012 in Miami. Richard Crowley came to the stage and threw the new term at the crowd and got back a few long faces in return. Richard’s own words, when referring to the Library-Oriented approach:

Package logical components of your application independently – literally as separate gems, eggs, RPMs, or whatever- and maintain them as internal open-source projects… This approach combats the tightly-coupled spaghetti so often lurking in big codebases by giving everything the Right Place in which to exist.

His talk was very solid and I recommend everyone with a hard-core-techie-heart to spare a few minutes on it. You can find his reflections about developing interoperability HERE.

It caught my attention just by the name, because I’ve been saying, “It’s like SOA, but with libraries” for some time now. “It’s like SOA, but with libraries” always came up when I was trying to explain an architectural pattern for building solid systems and frameworks. In general, LOA is just a way of thinking about software engineering. Library Oriented Architecture defines the structuring of libraries for domain ontologies and it has 3 basic principles:

  1. A software library implementation and subject area expertise must be constrained to only 1 ontology domain.
  2. A software library that needs to use concepts and artifacts from a different ontology domain than the one it belongs to, must interface and reuse the library corresponding to that specific ontology domain.
  3. All domain specific software libraries must be maintained and supported with separate lifecycles.

Before we get into the weeds here, we ought to ask ourselves: Why in the world do we need a new term, or a new architecture, or a new anything in software engineering? Well, we don’t, but if you care to write badass apps and software systems that can evolve gracefully with time, this can turn out to be a very good road to take. For those who enjoy bullet points, here are some of the motivations to explore LOA a bit further:

  1. Simplify configuration management of distributed systems.
  2. Build highly reliable software systems because of the inherent properties of the LOA principles.
  3. Increase the Maintainability Index of your distributed systems and integration repositories.
  4. Minimize the risk of high coupling, especially for large systems (read Writing Elegant Code and the Maintainability Index).
  5. Bring developers up to speed orders of magnitude more quickly than a traditional system. Move developers and teams across libraries and domain ontologies and collaborate seamlessly.
  6. Spot bugs and zero-in on the problem almost instantly. There is something to be said about the amount of time a developer spends debugging.
  7. Maximization of the Bus Factor of the software engineering team.
  8. Information Systems build using LOA are technology-independent, and have the ability to entire libraries and domain implementations with localized impact and minimal upstream ripple effect.

Ok, enough reading, let’s see how this materializes in a diagram.

Library Oriented Architecture

Note that this is a specific implementation of Library Oriented Architecture for compiled libraries. You can adapt this to your own needs for scripted languages and even mix it around however you want. For the sake of simplicity, we’ll stick to this sample for now.

The second thing I want to note here is that the diagram is not describing how to implement LOA. It simply lays the foundations for a software engineering practice that happens to follow LOA principles. I’m sharing this because I think is useful and maybe someone will like it enough to offer some suggestions to improve it further.

I want you to notice a couple of things that are illustrated on the diagram:

  1. All 3 principles mentioned above are followed.
  2. The framework favors convention over configuration. Lib names, namespace naming and schema conventions are noted in the last column.
  3. You can clearly dissect the domains vertically and they span all the way from the data storage layer to the actual library implementing the domain specific logic.
  4. A library representing an ontology domain never interfaces with the data-sources, or even data access layer, from any other domain; instead it interfaces directly with the library representing that domain.
  5. Services are merely wrappers of libraries, with minimal or no business logic other than the orchestration of the libraries it needs in order to fulfill its function.
    • This is important because services are always tightly coupling their technology implementations and serialization mechanisms (WCF, ASMX, SOAP, REST, XML, etc.)
    • Part of the service implementation concern is usually dealing with this technology-specific fuzz that is unrelated to the actual business functionality the service is providing.
  6. Exception handing is bubbled up to the lib layer, such that we always get meaningful stack traces when debugging.
  7. Logging, as a cross cutting concern, should be manageable at all levels of the framework, however the domain deems necessary.
  8. If the implementations of the domain-specific libraries share a common framework, such as .NET or Java, they most likely have a superseded library set that extends each framework. For the example illustrated in the diagram, we called them framework infrastructure libraries, or Common Libs for short.

So, now that we have a framework for engineering our software needs, let’s see how to materialize it.

Suppose you are working on the next Foursquare, and it comes to the point where you need services that help you normalize addresses, and work with GIS and coordinates, and a bunch of other geo-location functions that your next-Foursquare needs.

It is hard sometimes to resist the temptation of the ‘just-do-it’ approach, where you ‘just’ create a static class in the same web app, change your Visual Studio web project to make an API call to 3rd party services, and start integrating directly to Google Maps, Bing Maps, etc. Then you ‘just’ add 5 or 6 app settings to your config file for those 3rd party services and boom, you are up and running. This approach is excellent for a POC, but it will not take you too far, and your app is not scalable to the point it could be with a Library Oriented approach.

Let’s see how we do it in LOA. In this world, it takes you maybe a couple of extra clicks, but once you get the hang of it, you can almost do it with your eyes closed.

  1. The Lib Layer
    1. Create a class library for the GEO domain ontology. Call it something like Geo.dll or YourCompany.Geo.dll. This library becomes part of your lib layer.
      • Deciding the boundaries of domain ontology is not an easy task. I recommend you just wing it at first and you’ll get better with time.
      • You need to read a lot about ontology to get an idea of the existential issues and mind-bending philosophical arguments that come out of it. If you feel so adventurous you can read about ontology HERE and HERE. It will help you understand the philosophical nature of reality and being, but this is certainly not necessary to move on. Common sense will do for now.
      • Just don’t go crazy with academia here and follow common sense. If you do, you may find later that you want to split your domain in two, and that is OK. Embrace the chaos and the entropy that comes out of engineering for scalability, it is part of the game.
    2. Define your APIs as methods of a static class, and add a simple[sourcecode language="csharp"]throw new NotImplementedException("TODO");[/sourcecode]
    3. Write your Unit Tests towards your APIs with your assertions (Test Driven Development practice comes handy here).
  2. The DAL Layer
    1. Sometimes your ontology domain does not need to store any data. If that is the case, skip to step 3, else continue reading.
    2. Create a new library for the GEO domain data access layer. Name it according to the convention you previously setup in your company and dev environment. For this example we’ll call it GeoDal.dll
    3. Using your favorite technique, setup the data access classes, mappings and caching strategy.
      • If your persistent data store and your app require caching, this is the place to put it. I say if, because if you choose something like AWS Dynamo DB where 1 MB reads take between 1 and 10 milliseconds, maybe you want to skip cache altogether for your ‘Barbie Closet’ app :)
      • Memcached, APC, redis, AppFabric, your custom solution, whatever works for you here.
      • You can also use your favorite ORM (NHibernate, Entity Framework, etc.) and they already come with some level of caching on them.
      • Bottom line, LOA does not have any principle preventing you from going wild here, therefore your imagination and experience are the limit.
  3. The Data Layer
    1. For this exercise suppose we need to persist Addresses, Coordinates and Google Maps URLs.
    2. I suggest you scope your data entities by your domain ontology. A way we’ve found to work quite nicely is to use named schemas on RDBMS and setup namespace conventions for your NoSql databases.
    3. For the GEO domain schema, we used SQL Server and created a named security schema called [Geo]. The use of named schemas makes it easy to avoid long table names, provides nice visual grouping of entities and a more granular security for your entities.

When it comes to data modeling, another technique I like to use is that of unaltered historical event data. Any ontology domain can be dissected into 3 purpose-specific data models: Configuration Data, Event Data, and Audit Data. They all serve very different purposes and in general we like to keep them in separate schemas with separate security, this way we’re not comingling concerns. Each concern has a different DAL library and potentially they all interface with the library representing the domain at the Lib Level. This post is already way too long, I’ll try to cover some more data modeling strategies in future posts.

Now that we have a clearly separated domain library for our GEO domain, we can decide to wrap with whatever technology specific services we need. This is very convenient because when you want to move your SOA stack to a different technology, you don’t have to re-write your entire domain infrastructure, only the service layer. More importantly, it allows for greater scalability, since it degrades gracefully and plays nicely with different frameworks and technologies. A well implemented Library Oriented Architecture can be said to be technology-agnostic, and that makes it a great SOA enabler.

That’s it for this episode folks. Send me your comments or emails if you are using Library Oriented Architecture, or if you have any suggestions on how to improve the methodology or framework.

Happy coding!

2 Comments

Comment

The Myth of the Genius Programmer

http://www.youtube.com/watch?v=0SARbwvhupQ&list=PLCB5CF9838389D7F6&feature=view_all

From Google I/O 2009, here are Brian Fitzpatrick, Ben Collins-Sussman about the fears of programmers and the fear of looking 'stupid'.

Comment

1 Comment

Must-Have Visual Studio Extensions

I'm just setting up a new dev box now and there are always some things, like extensions and tools, I feel they are a must-have for Visual Studio 2010 devs. Some of these extensions like the Web Essentials have already made it to Visual Studio 11 (still in Beta), but I still wanted to share my preferences for my must have VS extensions:
  1. ReSharper
  2. NuGet Package Manager
  3. Productivity Power Tools
  4. Web Essentials
  5. Image Optimizer
  6. Javascript Parser

There is a cool Channel 9 vid where Mads Kristensen does a walkthrough of some of the goodies in his Web Essentials and the Image Optimizer. If you are in the web dev space, you should check it out: http://channel9.msdn.com/Shows/Visual-Studio-Toolbox/Visual-Studio-Toolbox-Web-Essentials-and-CSSCop

1 Comment

2 Comments

Configuring ReSharper Code Analysis

This article is supporting the article "Definition of DONE" with regards of code cleanup.

Having your code CLEAN is not only elegant, but also a good practice and habit to have specially if you work in a team environment. Every developer have their own styles, likes and dislikes when it comes to code formatting, ordering, grouping regions, etc; precisely because of this, is always good to get the team together, crack-open some Red Bulls and come up with your code standards and practices (and while you are at it, put them up on the Wiki). You can then use automation tools for your IDE to speed up the process and make sure everybody is generating and organizing their code files using the same rules. In my case, at work, we are living in the .NET world and ReSharper 4 or later in Visual Studio does the trick perfectly for us.

ReSharper has a feature called "Code Cleanup" that accomplishes exactly that: it automates the suggestions from the "Code Analysis" and applies them to code files. You can remove code redundancies, reorder type members and remove redundant using directives among many other things. This is not the only one of the many useful things ReSharper offers to .NET developers, but is certainly one I use the quite often. Let's see how it works.

At the time of this article I'm using VS 2010 and ReSharper 5.0, so all images are representative only of such software and version.

Make it perfect in one machine

The first thing is to configure all the settings in ReSharper on one machine and then share your settings. There is plenty to go through and each team may have different requirements. For example, by default ReSharper has the inspection severity on the unit "Redundant 'this.' qualifier" to "Show as warning" and in my team WE DO LIKE to use the 'this.' qualifier regularly, so we changed the R# setting to make "Show as hint". Things of this magnitude does not impact performance in code execution, the compiler simply ignores it; and they are more of a "style" type of inspection you can always tailor to your specific needs.

Once you have your Dev Box all configured, simply export your settings so the rest of the team can use them.

You can download the style we are currently using HERE.

That is the official way of sharing things using ReSharper, BUT... there is a small problem: that only shares the code inspection rules of the ReSharper settings and not other[important] things like the "Code Cleanup" settings. Breathe with ease ninja, there are solutions.

  1. If you have ReSharper 5.1 => you can use Resharper Setting Manager. This is an open source tool (CodePlex hosted) that allows you share ALL of your R# settings very easily. Officially you need R# 5.1 for this tool; I've tried it with R# 5.0 and have had a success rate of 0.5 (there is some black magic going on there that makes it work only half of the time with R# 5.0).
  2. Go old school and just copy the settings straight from the application data on your installation folder. Just navigate to  %appdata%JetBrainsReSharperv5.0vs10.0 and copy the files UserSettings.xml and Workspace.xml to your repository (or a share folder, or email, or... ). You can then get those settings from other dev machines and put them in their respective places (close VS first).

Now when you open VS on the other dev machines you'll find the same configuration and code cleanup settings as the original box. Sweet!

Using Code Cleanup

When using VS with a source code file open, the code inspection engine will always notify you of how well (clean) your code is doing with regards to your R# settings. The keyboard shortcut to run the code clean up is Ctrl+E+C or Ctrl+K+C depending on the shortcut schema layout you are using. After you run it, things will be looking better for you ;)

  • Red light means there is a BREAKING change on the code, and will probably not compile.
  • Orange light means the code doesn't have any breaking changes, but it can be improved. You'll get some warnings and some suggestions.
  • Green means you are good to go buddy. Ready for commit.

Part of our Definition of DONE is to get a green light on your source files. No exceptions.

You can find some more useful features of ReSharper like solution-wide analysis HERE.

2 Comments

4 Comments

On Hiring: One in the bush... too much to ask?

Today almost any tech enthusiast can say "I know how to program in C#" or "I know how to create awesome websites using RoR". With the evolution and abstraction of programming languages, it is true the knowledge required to make a program writing code on your own has become increasingly easy. My aunt, a 58 year-old catholic woman with 5 children and 8 grand-children just made her own website using nothing else than Dreamweaver. The point I want to make is "with popularity also comes mediocrity". There are a lot of self-titled software engineers in the job market that cannot tell you what a pointer is, nor can explain the basis or recursion. Many of these people make it their goal to memorize and learn a specific IDE, framework or language and then they themselves software engineers. Hey, when I was a kid, I built a tree house with my friends and... IT NEVER CRACKED... I'm a genius! I'm going to apply for an architect position to build the next Trump Tower!

As I'm writing this post, I'm going thru the process at work of interviewing some candidates for 2 openings, one as a Junior Software Developer and another as a Software Engineer with substantial experience and hands on project management. I'm living a nightmare. This is the second time I've had to hunt for people to add to the team, and I'm surprised on how many applicants are out there applying for positions they really cannot fill. After almost 10 days reading resumes, making phone calls and talking to many applicants; I've only been able to handpick 1 out of 57 applicants to move on to the second part of the hiring process. The CEO of the company (my boss) says I'm too rigid with the requirements and too tough with the applicants... I don't know if the expectation is to lower my standards (hard for me) or keep hunting for the right person and sacrifice the lost time for the quality and productivity the jewel will bring when it appears. In the meantime I can keep looking through a pile of stupid resumes full of lies and incredible fictitious accomplishments. This is the process I'm going through for the hiring process. As I said, one position is for a Junior programmer and another one for an experienced engineer.

  • Filter 1: Job applications go first to another guy that filters out the obvious mismatches based on non-technical stuff, such as their VISA status, ability to speak English fluently and so on.
  • Filter 2: Then, this guy sends the resumes to me and I take out those that are really really obvious mismatches based on their previous experience and field of interest. Here is kinda crazy what you find; people that graduated in 1976 from computer science and for the last 20 years they've been a manager assistant at Walmart. I mean, seriously?
  • Phone Interview: At this point we are down about 50% of the initial applicants. Then I order their profiles by the "interesting" factor and start calling them in order. I go through the same script with each one of them, and try to cut them short when I see is not going to work. Here is my script for the phone interview.

------------------------------------------------------------

  1. Chat about the company, what we do and what type of job the candidate is going to be doing.
  2. Chat about the candidate. Get him/her to relax an feel comfy talking to me.
  3. Question about most recent project he/she worked on. Specifics.
  4. Three questions about OOP.
  5. Three questions about programming languages (references, heap and object creation and disposal).
  6. Three questions about .NET Framework essentials.
  7. Are you satisfied? (the applicant)
  8. Questions and Answers.

------------------------------------------------------------

  • In-Person Interview: If the person passes the phone interview with a positive outcome, then he/she is invited to the office to continue the screening. An appointment is set.
  • A Day at the Office: the candidate comes to the office. I make sure to ask him/her the same questions I thought they could've been answered better in the phone interview. I chat with them a little bit and introduce the candidate to the team members and people they'll be working with. 2 or 3 of the software engineer team members will actually do like a little personality screening in private with the candidate with the goal of helping me determine if they think it'll be a good addition to the team. They can either give me thumbs up or thumbs down and tell me why they think it may or may not be a good addition to the team. I value a lot the criteria of my team, ultimately they are going to be working together and nobody wants a week link or a introverted soul in the group that doesn't know how to play team sports.
  • The test: After the office showing and team interview and more questions, it comes the written test. IT IS INCREDIBLE HOW MANY PEOPLE WILL PASS THE PREVIOUS PHASES JUST TO FAIL HERE. The test is usually something very generic but that requires a minimum knowledge required for the kind of job we do (and that they are applying for). This time around the exercise is to create a remote application HTTP accessible with a function that, given an integer it returns the second consecutive prime greater than the integer. The second part of the exercise is to create a separate client application that uses such method and provides a minimal UI to use it. I give them 45 minutes to complete and another 10 minutes to explain to me what they did. So far none, not even 1 candidate have completed the such simplistic task in 45 minutes. DAMN IT! I've encountered myself explaining to recent grads and candidates with Masters what a PRIME number is!!! BOOOOOOOOOO!!! WTF is going on?!?
  • Negotiations and Hiring: I've only considered 1 person as a possible candidate so far to fill in the Junior position. Negotiations are straight and very much depends on what skills and drive the candidate shows during the interview process. If I'm sure you are the right person for the job, you'll leave the office knowing that and with a competitive offer in your hand. If you are kinda there, but never got to convince me about you, BUT made a strong point about your will to learn and future, I'll put you in the back-up bucket. The rest of the world will receive an email with a sorry and a good luck on their job search.

At this point there has to be something I'm doing wrong. I cannot believe there are no worthy software engineers out there looking for a job. Is it that the interview is too tough? Am I expecting too much from the marketplace?

4 Comments

1 Comment

SharePoint Site Architecture

In this post:

  • WSS vs MOSS
  • SharePoint site architecture

SharePoint is an ecosystem of software and server products as it was described in this previous post. To understand it better let's look at the architecture and object model that it is offered with SharePoint. The basic component of SharePoint is Windows SharePoint Services (WSS from now on). WSS is a free server component that must be installed in a Windows Server OS and contains the basic framework and infrastructure that supports any other SharePoint offerings like the different Microsoft Office SharePoint Server (MOSS from now on) versions.

You can think of it the same way selling cars work. You get the basic model with a chassis, 4 tires and a steering wheel (that will be WSS) and then you can add stuff to it like a great sound system, a dynamic design, GPS and Cruise Control, etc (those nifty things being the analogous to MOSS).

So now that we have an idea of how WSS relates to MOSS, let's see exactly how their offerings stand apart.

As we can see all MOSS trims are nothing more than a very useful set of extensions and additions to the WSS platform, offering users a more powerful and pleasant experience out of the box. These extensions are supported thanks to a very complex architecture and framework that has evolved quite a bit over time.

In a nutshell the industry refers to a SharePoint installation (be it WSS only or MOSS) as a SharePoint FARM. A Farm is a collection of physical SharePoint servers  that serve the content to users. A Farm can be formed using different topologies that we'll explore later, before we get to that, let's actually see what is going to be served, how does SharePoint structures its web sites and content.

Farm:  A collection of SharePoint Web Applications.

SP Web Application:  A collection of Site Collections.

SP Site Collection: Formed by a single root site (aka Top-Level site) and a collection of sub-sites.

SP Sub-Site: A collection of pages, lists, libraries and content that are displayed through the browser.

Each of these components in the SharePoint site architecture have different rules and security settings and restrictions. As you can see, this is a tree structure and very successful way to isolate users and content to ensure confidentiality, availability and integrity in the data security. This model is very user friendly since everyone can understand a tree structure and most people are already used to it.

I always like to compare a SharePoint Farm with a physical hard drive. If the Farm is the hard drive, then we can think of the Web Applications as partitions of the Farm. Likewise Site Collections will be Folders directly on the root of the partition. Sub-Sites are also represented by folders that only go inside of the Site Collections and so on, you get the idea.

Because is easy to understand the site architecture SharePoint offers, users can quickly provision (a fancy word for 'create') web sites with the click of a button. Because of this, planning for a successful deployment and IT infrastructure to support all the greatness SharePoint offers does not come at ease. Planning the right topology that will hold all of this together is very important for a successful SharePoint deployment. More on SharePoint topology and planning in upcoming posts.

1 Comment