SharePoint and large files

Suppose you want to put SharePoint to some good use and you or your CIO thought “let’s just use SharePoint as a DMS”.

What can go wrong here? It can store files, users love it, we have plenty of licenses, integrates into Office, has search, versioning, metadata… all sorts of neat stuff.

This is completely true and SharePoint can definetely act as a very good DMS. If you know its limits and really understand that theoretical limits are a different beast in real life. As a side note, a DMS system is almost never just a neutral tool. It needs customization (or at least heavy configuration by a specialist) to implement business rules and processes. I’ll maybe touch on these on another post.

First, you will want to check the official Microsoft Sharepoint boundaries page here. Even in “small” DMS environments you will need to pay close attention to the “File size” limit and to the “Content database items” one.trextrying

So, you’ve found out that the maximmum file size is 2GB and that the recommended file size ranges between 50 and 250 MB. Chances are that you have many important files bigger than this. Jump to the internet and the best MSFT advice you’ll get is somehow sommarized in this TechNet post. Which basically says that you can’t. Or shouldn’t.

Don’t get me wrong. Technically, anything is possible in software and there are convoluted ways of making SharePoint handling files over 2GB in size. Such as splitting the file in volumes and storing those instead… and aggregating them on download… messy stuff.. Or use RBS for offloading the content database from SQL Server while you aim for the 4 TB limit (but still helpless for 2GB+). After which you are forced anyway to restructure your core DMS business solution logical architecture.

I, on another hand, have the opinion that a technology platform (eg SharePoint) should help me concentrate on business solutions (eg: solve DMS problems) not require advanced expertise just to make things kinda’ work and then keep an eye on the business requirements and solution to not break my pretty little technical-very-backend architecture limitations.

Back to the issue: can it be done in SharePoint?

Yes. But we need also another content platform to handle the exceptions.

What we can do is build a custom document library (or let’s say, a new type of document library with our additional features) which enable content transfer to and from the user using another content platform for storing the large files. This can work also to overcome the limitation of the number of items in a document library/content database.

Features can be implemented to create/destroy native SharePoint items as needed so that the user will be able to use SharePoint standard document library features on some selected documents if they really need to (provided that it’s not larger than 2GB, case when you are stuck with upload/download/stream only for the content.. no inline editing for you).

This way, the SharePoint limitations will not apply anymore (since the features are presented “at the glass” and interact directly with the other “hidden” content platform) but you still have the SharePoint features for selected items should you wish to bring them into the native space. Searches can be done a variety of ways. Either using SharePoint features (since it can index external stores) or the external CMS may have its own API for that.

Security is also ok, since you will choose a content platform which can really do item level security even for many or large files (in SharePoint you are advised to not use item level security if you have many documents in your library).

Linking the external CMS underneath a SharePoint document library gives you a lot of advantages… I’ll let you discover those, this is just a blog post.

One more hint: if your chosen content platform exposes itself as a CMIS provider, then you really hit the jackpot. Strategically speaking, because in real life SharePoint cannot act as a CMIS client anymore (since ShP 2013, although it could in 2010). But I think they will not be able to ignore this in the future and anyway you’ll find partners developing ShP addons to expose CMIS in the UI.

Here you have it. I just shared with you our solution on how to make SharePoint as a DMS when considering its large files limitation (and also works for the “many files” limitations).

Documentum underneath SharePoint?

I’m sick and tired of this discussion.SharePoint and Documentum integrated together

Every once in a while I see a new post / comment / topic /spam around how great would be to have SharePoint as a UI or for collaborative content and to use Documentum “in the back end” as the repository of choice for compliance, bpm, records or whatever other “complex ECM” thingie. I don’t list the last 10 seen this year already in order not to offend their authors.

We’re talking about this for many years already. More than 3, I think. Do you see it happening? Not. Otherwise the discussion would not be “how great would it be” instead of “look how great it is”.

Will it happen? Probably not, and this is my take on “why”.

I’m talking about the “true” integration, in which the content lives and can be manipulated in both systems. In a business scenario, not just in OOTB “platform” features.

Think “security” for a moment. This is one major pain the integration must solve. Synchronizing the both worlds would be an awful task.

Think business rules (pervasively to be applied in both systems). Yes, there are BRMS products out there, but the large majority of ECM customes don’t have any such thing.

Think associated information bypassing the boundaries of the system (eg. annotations made in one platform to be available on the other).

Who needs to solve this problem? The f.cking solution provider. ECM is not installing itself (contrary to some belief, especially in the ShP world).

What’s the incentive to the solution provider to make an integration so complex and using two very different kind of technical resources? Nada, usually. The provider will focus to deliver a solution with a minimal pool of resources and with the less possible risk factor. Microsoft + EMC in the same boat, with no or very little experience/knowledge on how this goes in reality? This generates risk+++

What’s the incentive to the business owner to ask for this? Only to justify the purchase from both license goliaths. There is hardly a true business case around it. What can be done in ShP can also be done in Dctm and vice-versa with mostly the same cost. It’s a “political decision” where to do it, actually. Or a decision simply taken by someone in the team and swallowed by the rest.

So, while the community (mainly the technical area, let’s face it) dreams about pink elephants, the business oriented SI/… selects one platform and implements the solution on it. If forced, it can move content around from Dctm to SharePoint and back… but that’s not integration… is it?

Sure, it’s nice and handy to have an RBS connector for Documentum. Or to have some webparts exposing various Documentum features. But this is all we can get and apart for some simple use-cases, they’re… oh well… simplistic.

Building an intranet portal on SharePoint 2010

Our intranet was merely a SharePoint 2007 installation with several document libraries and lists and some random department sites. The look and feel was pure OOTB SP. So, some months ago we came up with this crazy idea of changing the status quo and building a new and real intranet portal.

We did this by starting an internal project. I don’t know how internal projects happen in your organization, but as I’ve seen my fair share until now I’m guessing it’s not much difference. An internal project is always “least pirority” for almost everyone involved. And it’s hard to change this as they are not perceived of generating any money (vs. any external customer).

With this in mind, we’ve set it up. Marketing was the project owner, somehow together with HR (“internal communication” is handled by Marketing in our world). We had a project manager / project sponsor (me). Each business division and department had a unique representant in the project, forming the full team of about 25. And there was also a core project team of about 6 (IT, content, design, strategy) to do the main work.

Here you can find some of my lessons learned from this project.

1. Make sure you know the organization

The person who leads the project need to know and understand the organization in & out. Building an intranet in which you aim to touch all departments, and generate valuable information for anyone in the company is a very complex undertaking. Especially if the company is quite large with multiple business lines and agile in day to day operations.

If the PM does not have this knowledge (say, s/he’s new to the company) then there must be a key project champion in the team which has it. And they must work very well together.

We scored very well on this. I’m with the company for many years now.

2. Decide and move along

We had a big team (imagine having status meetings with 15-20 people, some from management). You can safely assume that if 2 persons are involved, they will have 3 opinions. So, decisions need to be made by somebody and followed through afterwards.

3. Prepare the organization and communicate progress

The intranet has impact on almost all organization members. They must be informed that a new one is coming and expectations set correctly. This is also a motivator down the line to keep the project running and progressing, otherwise people will start wondering what happened.

4. Always keep an eye on the value to be provided

When building an intranet you will encounter many things which can or should be done. Always keep focus on the value (best is to make a checklist) which you aim to provide to the organization. Eg: enable people search, help people find their way through procedures, show the news, enable corporate wide communications, provide links to all relevant apps, provide a valuable search service (enterprise search with facets is a must in today’s day and age), integrate app XYZ…

It’s easy to sidetrack to building more and more web pages with content which would most likely mean nothing to 90% of the organization but is cool or nice for 5%.

5. Identity

The intranet should have an identity of its own. And must have a coherent design. Coherent with the company identity guide (if you have one, please tell me you do). My opinion is that it helps the portal to have a name of its own. It makes people adopt it better, imo.

6. Content

An intranet is nothing without content. If one can make do without fancy “Web 2.0” stuff, it cannot succeed with an intranet without content. Users come to the intranet portal (voluntarily) for at least these reasons: find info about a person, find info about a department (eg: find what that department role is in the organization and who to contact from there), find document templates for day to day operation…

So, plan to allocate around 70% of the effort to the content creation part. Recognize that content cannot be generated by 2-3 persons, but needs to have this task distributed alongside all department representants. Make sure each of the department representant has the authority from the department head to collect and present the related content (this is a must, since sometimes departments will nominate juniors for this task).

How you organize the team to build the content in preparation for the portal, it’s up to you. One of the simplest ways is to make them collect the content in word / xls / ppt documents and put them in a folder structure resembling the final portal structure. Not so fancy, but easy to understand and do. And can be done in parallel with the technical implementation of the portal.

Please don’t forget the existing content (which may lay in the old portal, some fileserver…). It needs to be migrated. Plan, monitor and act. As in any migration… test test test.

7. Structure

Define a portal structure. This is where everyone should brainstorm to and then one needs to make a decision. Try to follow these guides: any somehow relevant piece of information must be no more than 2 clicks away from the main page, the main page must be clean and small, similar organization units must have almost identical portal structure (if someone enters a department/project/division/… page, it should see mostly the same sections as in other departments/projects/division/…).

8. Functionality – get a grip on technology. Early.

You can spend a lot of time building content and imagining the portal structure, but nothing will prepare you completely for the moment you face “the technology”. People who make content don’t really know what the technology can do for them. Even IT doesn’t know, believe me (we’re talking about SharePoint 2010 here).

We did it like this: define the structure and a design prototype. In these discussions the SharePoint specialists are involved also (even if the talk is mainly about business content and rules). Afterwards, put up a playground SP install, build some content templates, page layouts (based on the design), basic navigation structure… and let the content authors loose! You will find at least 5-6 of them (from 20+) who are eager to try and don’t mind that they will need to re-post the content several times. In this manner you will start to define your content editing and presentation rules. Ah, by the way: train the poor content editors before you leave them in ShP. A few hours of informal training will do wonders for the project.

When this playground time is done, you can gather all business representants, show them the intranet prototype, discuss the content rules and tell them to go to work: move their already prepared content into SP.

9. Have a passionate technology guy/gal

Only the SharePoint 2010 expert will know what he/she can do with it. The  IT in general will not know.  Why do you care? Because if you just treat SP as “yet another portal tech” you’ll end up with a dull intranet. SP has some great features but you need to know about them and use them in the business context.

So, make sure the IT guy responsible with implementing it is a heavy supporter of the project idea and wishes to innovate. Make sure he participates in all the project meetings and listen to the business explore ideas and requirements. You may need to pull him from the comfort corner where IT staff likes to stay, but it will be for the good of all.

10. Make sure you know what happens after go live

So now you’ve done it. You just created a living thing, under daily scrutiny from all the company employees.

Please make sure it works well (plan and act with IT to have proper maintenance and helpdesk in place) and provides continuous value (define responsible for recurring tasks to update the content, put in place procedures to add new information when certain things happen).


So,  what’s this got to do with SharePoint 2010?

Well, we did it with SP 2010. I must say, I find it not very well-rounded. It serves the purpose but has some idiosyncrasies of its own.

Some good things:

  • integration for the FAST engine – makes wonders for reviving that old search feature you had in the portal and about which everyone bitched about.
  • my sites – gives employees a space to describe themselves
  • content editing – with the proper templates in place, business owners can understand it and work with it ok

Some bad things:

  • layout limitations – you have to go though many hoops to apply a not ordinary design. Or you need to make concessions on the design
  • mobile access – terrible. On iPhone/Android/RIM…. almost forget it. IPad is tolerable (since it’s a full browser exprience more or less) but still doesn’t always work properly

Am I happy with how it went? Yes.

Did it cost an arm and a leg? No.

Can it be done better? Much better.

Would I recommend SP 2010 for an intranet portal? If your IT is Microsoft oriented and has in-house SharePoint skills – yes. Otherwise, it’s a long story.

Documentum or SharePoint – part deux

8 months after the initial post on this topic… And a few more projects on SharePoint and Documentum alike…

I have 2 big enterprise customers which are both Documentum and SharePoint users. Both got from their CIO the directive: “SharePoint is good, it is fantastic, I want it this year”. Now the year is ending.

It was an year in which my Documentum projects at those customers came almost to a stall. Not completely due to the SP frenzy, but that did indeed factored in heavily. I was also involved in some SharePoint projects (yeah… frenzy mode…) with them, so overall business was ok.

But, what’s the experience at the end? Did the customers go live with SP? Yes. (or else it was their lives 🙂 )

How was it? Ugly. Plain ugly and filling all teams with frustrations, building rivalries and making IT crazy.

Why? Lack of experience.

Deploying something enterprise wide is difficult. SharePoint makes it more challenging because of at least these factors:

  • SP has a totally different architecture and feature set/names/capabilities than traditional ECM and collaboration platforms. You can’t reuse skills or practices. Only the most general knowledge will apply – and that’s not enough.
  • SP is not Enterprise grade. It’s marketing slides and lobby actions are, but deep inside it’s still a departmental tool. The scalability topic and distributed content/installations… the many things under this hat (including best practices) are simply missing or strangely addressed
  • There is a serious lack of true professional expertise in the market. Many can do the talk (eg. MS Services) but can’t do the walk.

And this is normal, don’t get me wrong. All the above are and were very clear to any architect worth his money. But strategies are not done by them, are done by CxOs.

So.. is Documentum better?

For all the above reasons yes. You can count on the old dude to deliver and you can for sure find expertise in the market. Why? Because they walked on this earth longer than the SP specialists. And they have seen many more projects and been through more real life situations.

Is this enough to say Documentum is better? Well… for 2010 yes. Maybe also for 2011. At least in large companies or for non-trivial content management tasks.

I surely trust the MS ecosystem (me included) to learn from current SP experiences, push forward and finally achieve the maturity needed for implementations. My feeling is that it will happen around the end of 2011. So I think Gartner is off by an year when they said in the latest MQ that MS rulez the ECM world. Come on guys, even you don’t believe this… marketing and hearsay is not delivery…

Where will Documentum be at the end of 2011? Not far away unfortunately. Customers need to be conscious that a major evolution awaits in the Documentum landscape. It’s present offering will be completely transformed in the 2-3 years to come (and this is close term in corporate speak). How do i know this? I don’t. I feel it and lie to myslef that I can read between the lines.

Let’s evaluate the main Documentum product roadmap:

–  Content Server  – to be completely transformed (say hello to Information Server). This means DFC should be gone. Big hit. Win or lose on this one for many partners or customers.

– Webtop – ye’ olde’ end user interface is going to retirement house. It will be missed. And nothing mature enough to replace it yet. And EMC proven itself to need 2-3 years at least to mature a UI. An extremely large number of customers use Webtop or a derivative solution which took years to build and mature. What to do with that? Sure… revenue for partners… but who’s willing to pay this?

– TaskSpace (aka. xCP) – based on the Webtop SDK, will need (and will sure have) a rewrite. Oops! That doesn’t sound good. Another 2-3 years to mature. I don’t know what will happen to forms built in Form Builder.

– Centerstage – should have been the new UI. Currently is more of a playground. Plagued by stability and scalability, it needs at least 1-2 more years to reach the needed level.

– Process Server – I always wondered why there is such a product when in fact is the Content Server itself with some 2-3 additions. This will surely need a rewrite once the new Information Server arrives. There goes the core of xCP.

So… Which is better?

Any of them you can make them work in 2-6 months and deliver ROI in 1 year. If not, lay low and let them transform a little bit more.

Documentum or SharePoint

Eternal question for a large part of my customers.

Recently, this question turned into “Why keep Documentum if I have also SharePoint?”. At least for some.

Really… why keep it? When is Documentum better than SharePoint for an organization? It’s sure they can coexist, but are they both necessary?

I try to answer this without looking at the existing deployment of Documentum in such organization. This can be sometimes easily changed with a flick of a top management decision. Bad experiences can help also :). So let’s look at the future.

Limiting this further, let’s not talk about transactional content management. Let’s do an evaluation more on the so called “Information Worker” area. EMC purists will say “but since you are stripping away such pieces you’re dumbing down Documentum and it’s not fair”. Well, nothing is fair in love and war. And I’m pretty sure that if an organization would have a massive need for a transactional content management scenario it will evaluate it also from EMC side.

So, looking at Information Worker.. what can Documentum bring to the table?

Core DMS stuff… it’s already pervasive. Folders, files, documents, metadata, security, versions, light worklfows… All this is already in SharePoint. Don’t mind the small differences, they don’t count in this war.

Renditions? Yes.. this is a cool thing. but how to turn in into business advantages? The business is paying usually, not IT. So we need to explain this to business. Can’t. So it’s a no use feature for us. Moving on…

Virtual documents? Sorry. The concept is “so 90’s” (to quote some teens). I have yet to see a regular company making use of VDM and loving it. And we can pretty much implement related features in SP 2010, some ootb even.

What’s next.. storage management. Yes. This is really a differentiator. How to explain it to business? Tough… without a clear value proposition/great sales pitch  this is doomed to be just a bullet in a powerpoint slide.

Cool API (DQL/DFC/DFS)? I love them. Business couldn’t care less. Another bullet in my ppt.

xDB. Eh… another very very nice thing. Why should business care? Well.. we need to show a real use case. From their processes… show some ROI… This really needs to be brought forward since it has no match in SharePoint. Could be a winner, but still requires a partner application / specific solution / CEVA.

Ease of installation? Forget it!

Ease of maintenance? Who cares? And anyway, an existing customer would know everything about the ever troublesome Indexing Server/Agent, UCF and  Java Method Server.

Ease of deployment for customizations? Anybody with some multiple attempts to do DAR installations would grin. Show me how to record a custom metadata information in Documentum… sure.. it’s simple, install Composer, put your development hat, tinker in some “show more” menus, do the technical mumbo-jumbo, create the DAR archive (wtf!), go to the production server console (!), install DAR, should work. In SharePoint? Create new list. You see it works. Sure… it’s not the same outcome, but does the business care? Can you explain the differences? Who cares is not “properly” managed?

Plugins? This is good. Could bring you some leverage especially for surfacing / migrating old content from massive legacy repositories.

Aspects / BOF ? Not useful. In business talk, I mean.

Time/cost to develop something new?  Hmm.. come on! How many EMC Documentum knowledgeable people you have in your geographical area? What about Microsoft partners ready for a piece of SharePoint?

User interface? Sure… let’s change Webtop a bit for some users.. Call in the developers 🙂 Why don’t you use CenterStage? Uff… should I really answer this question?

What else? Search? Come on… I’m never sure what results the search will bring to me (remember, I’m taking about customers which already have and experienced Documentum for some years).

Upgrade to major versions? What about all the Webtop customizations I did? Well… you shouldn’t! 🙂

Why should I use Documentum? Only because I can see multiple business applications on it. But do I see them?

still thinking…..

This is big

If we can give credit Microsoft for anything is for being able to take a success and turn it into a bigger one. Yep, after reading some more things about SharePoint 2010 I’m more inclined to see a new wave (tsunami could be a better word) heading for the ECM shoreline.

Currently SharePoint has an well established foothold in the minds of most CIOs and IT teams across the globe. Some ECM professionals looked at it initially from a somehow superior position.. something like “whow, you’ve come to play with us… how cute…”. Now everyone in their right mind thinks differently. SP changed the ECM space and it’s coming back for more.

Some new things which I identified in the new release and are really a giant leap forward:

– unique document identifiers. across all the SP “sites”. Until now sites were quite isolated. No more – one limitation gone.

– metadata based navigation. This is required in all the projects I’ve did on ECM until now (100+). It almost always required coding (not configuring) or a sepparate licensed product (eg. Content Intelligence Services in EMC’s world). In SP 2010 is ootb and seems to be across the featureset.

– logical groupings of documents. Case management and many other ECM use-cases requires to have documents grouped together in a virtual “stuff”. Not a list, not a folder, an entity of it’s own. EMC Documentum had an old-time response to that: virtual documents. Quite clumsy and it always required customizations to make it really effective for the user. Now SharePoint has not only this but also thought of a visualization UI different than from standard items (yes, a grouping of documents should not behave and look as a single document… revelations 101)

– records management. You, traditional ECM guys out there… admit that you laughed when you heard SharePoint 200x did Records Management ;). I did. Come on… it was so inflexible, so isolated from other content areas… and performance… let’s not talk about it. No more! now it has a content dispatcher based on metadata, multi-stage retention policies, hierarchical fileplan, in-place record declaration… and even more… in place records management (doesn’t move the document just because it is declared as a record). And can declare almost anything as a record (e.g. wiki, blog…). Declaratively they also worked a lot on performance. Second thoughts, anyone?

Simpler (MS style), better, building on a success. How many ECM vendors can match this momentum (sic!)?

Looking back in 2009

When 2009 started we had the usual project line-up.

One particular project emerged in a nice CEVA solution on top of Documentum for technical documentation management. The most special thing about this project was that we encountered not no many technical challenges but many project management ones. We had our fair share of problems, but the customer PM was one of those persons who are afraid of taking any decision and this toghether pushed the end of the project to near the end of 2009. Anyway, the result is a very very nice application on top of Content Server, including a very nice own developed Silverlight image viewer to display huge drawings over a limited network. Lessons learned? Many!

The next project which comes to mind is one which had the official kickoff just before Christmas 2008: a BPM project for the biggest bank here. a project to manage all documents, regulations and other stuff which need management board approvals. the challenge, as stated in the RfP: “implementation time: 1 month”. Lol. Tried to educate the customer and brought that to about June. This was the full Documentum BPM suite to be customized a lot and then implemented. We know from the beginning that even June is optimistic. 1 week ago was the last go live. Err… there is some more tinkering to do but… almost done. Lessons learned: never customize Taskspace.

The rest of Documentum projects were in the regular key. One had a particular innovation when we needed to integrate Documentum with another external store device, not Centera. Built the connector, works like a charm. Now the connector is also certified as designed for Documentum. Yey!

Sharepoint wasn’t the bing bang we thought it will be. Small projects, rushed, etc. I guess the customers are no yet understanding the fact that if you want SP to play in a way which is not out of the box you need to pay through the nose for the development effort. Not to mention that in many cases you need to trade off future upgrades to needed functionality.

And i finally saw the first big FileNet project. DMS, worfklow, capture.. one nice piece of requirements. Integration with the customer ERP.. etc. I must say that the default client of FileNet (Workspace) looks much better than Documentum’s Webtop. And behaves nicer. What i don’t get is the internal data model representation. Looks awful to me, i need to see how it behaves for large impementations (may objects, different schemas).

We also embraced Adobe LiveCycle and Connect. The next projects to come in 2010.

The economic downturn affected us quite a bit – especially the cashflow, not the project pipeline. But we managed to hire new people and not reduce paychecks.

We also launched a great Data Center, one of the best in the country. SaaS, Cloud stuff… the works.

So, 2009 was overall ok. Technically very demanding and opening new prospects for 2010.

See ya!