Sharepoint Timer Job Paused

I have created couple of custom timer job and it was working as expected without any issues.

One day when I start the job manually (since I need to pull the data other system and push it to sharepoint list, which was as needed), and it started running without any issues.

But after couple of minutes it started showing paused along with other OOB timer jobs.

This seems to be more interesting why all of sudden all the jobs went to pause state.

Then I started looking all the jobs carefully and I can see there is job called "timer service recycle" which was alone running(couple of instance one in WFE and another one in APP).

Then I started looking into more details about this timer job and what exactly it will do is very well documented in below blog.

Timer Recycle Job

This job will start @6AM in morning and it will run for approximately 11 minutes in the farm.

Once this job finishes automatically all other jobs will start running automatically.

The Execute method of job definition Microsoft.SharePoint.Administration.SPUsageImportJobDefinition threw an Exception

Background:

We are using sharepoint 2013 environment with SharePoint Server with Enterprise Client Access License with SP1 installed(15.0.4433.1507).

Recently there was windows  update which was installed in this Farm(Test instance) and  after that there were couple of issues which started throwing.

In the event viewer I could  see lot of errors with the following error message

The Execute method of job definition Microsoft.SharePoint.Administration.SPUsageImportJobDefinition threw an Exception with the EventID: 6398

Since this is critical error type and Scom started triggering the mails very often.

When I check the error message its saying Access to the path 'C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS' is denied.

This seems to be strange behavior, the account which is running the timer job is part of local admin group which is having full access to the folder path.

Initially I assumed this was due to Diagnostic Logs which is throwing this error but I when I checked it was pointing to D:\Logs folder which I was configured initially and also if that is the case then it should not log the error in periodic basis.

Then I verified the Configure usage and health data collection log path, this was the culprit which was pointing to this error. Later I change the path to D: then all the error stopped logging in event viewer.

But now the question is why it started giving this issue after windows update.

I have raised case with MS to know more about the issues and will update once I get any reply

Sharepoint Search Error 0FF1CE15-013A-0000-0000-000000000000 -Error loading type library/DLL.

I was setting up the sharepoint 2013 server and configured the sharepoint search in one of the app server(1 WFE, 1 APP and SQL server)

Everything was working as  expected, but last Sunday our infrastructure team has done windows update in APP server and SQL server(it seems to be other update as well).

After that there was strange behavior in the farm, like while running the jobs it was throwing access denied error, health usage report etc. So we asked infra team to revert the APP server snapshot to before update installed.

They have reverted the snap shot of the server and once they hand over the server, I checked 
the all the service application which we configured earlier, and Search service application was crashed due to server update happened in the APP server where search is configured and I was getting below error when I open the search service.

The Execute method of job definition Microsoft.SharePoint.Administration.SPServiceInstanceJobDefinition (ID a41c8bc6-6ee7-4039-b171-277686fc0b96) threw an exception. More information is included below.  Unable to cast COM object of type 'Microsoft.Office.Server.Search.Administration.MSSITLB.CGatheringManagerClass' to interface type 'Microsoft.Office.Server.Search.Administration.MSSITLB.IGatherManagerAdmin3'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0FF1CE15-013A-0000-0000-000000000000}' failed due to the following error: Error loading type library/DLL. (Exception from HRESULT: 0x80029C4A (TYPE_E_CANTLOADLIBRARY))."

I thought my search service got screwed up and I need to create it from scratch. But I don't want to do that since everything was working as expected, so I just wanted to deep dive into the issue with ULS Log, Event viewer.

There was one clue was available in the error message i.e, 0FF1CE15-013A-0000-0000-000000000000

When I just saw this id, I assumed its  some thing related to Registry entry, either its having issue while accessing the Registry key or key might be corrupted(I was under assumption that whole registry got corrupted, luckily that was not the case)

Then I opened the Registry 

1) windows + run, type regedit it will open registry application(Make sure you open the registry with admin access else you wont be able to open)

2) Search for this specific key in the registry - Note here flower brackets are very important while searching else search return multiple value

3) This will be located under HKEY_CLASSES_ROOT\Interface folder

4) There will be 3 folder under this as like below

Above pictures shows the path of the key.

5) Here you need to select {0FF1CE15-0001-11B0-0000-000000000000} again curly braces are important and search under the registry folder HKEY_CLASSES_ROOT\TypeLib\

Please follow the path as depicted in the below screen shot.

As  we can see here its having the value "C:\PROGRA~1\MI95D8~1\15.0\Bin\mssrch.dll\1" which we can say its corrupted or its not having the proper location path.

This key value we need to change it to "C:\Program Files\Microsoft Office Servers\15.0\Bin\mssrch.dll\1" and save the entry.

Couple of notes:

1) These changes needs to be done in the server where search is configured, in my case I have done it in app server

2) After registry changes its recommended to restart the machine, please plan for that

3) After this registry changes search service  application will loose all its crawl history, need to run full crawl(basically search will not be in working condition, so we need to run full crawl)

4) In my case I have not created new search service application, existing application worked without any issue.

5) Query processing component will throw error if we search for items, in that time its required to start below services

• SharePoint Server Search
• SharePoint Search Host Controller
• IIS  Reset in all servers(Optional)

Loading this assembly would produce a different grant set from other instances

Recently in our test environment we encountered this error when we tried  to open the site.

I thought it might be some cache issue or db might got locked for that specific site. So I tried to open sharepoint central admin site then it started throwing this error.

Just before 5 minutes all the sites were working and after that suddenly it stopped after I deployed one of the dll to gac and did an iisreset.

I thought it might be due to latest version of the dll its breaking and I reverted the older version of the dll, but still I was getting the error.

Then I started trouble shooting issue by keep on checking one by one, is there any windows update installed recently, but there was no updates.

So service packs. no language packs no windows updates. Then I check the control panel to see is any other tools/software installed recently, luckily there was tool called Microsoft monitoring agent which is used for SCOM alert.

Our admin team have upgraded SCOM 2012 to SCOM 2016 which is culprit in my case.

Why this will break the application/sites?

Yes why this should break the application. As everyone is aware sharepoint is very complex and brittle in nature. It doesn't allow any other assemblies or any other tool to come in its way.

In our case this monitoring tool tried to inject their assemblies in sharepoint worker process and it caused all the worker process to broke.

In order to fix this issue I asked our admin team to uninstall the SCOM client and install it manually with NO APM=1

This exercise needs to be done in all sharepoint server where its installed.

If any one of you faced this issue and googled out there are plenty resources available which suggest to update registry entry or update web config file of the application.

All the above workaround doesn't work and even if it works we are loosing some thing which we are not aware what might it will break in the future.

Better to avoid all those and tried to verify if any tool/software has been installed recently based on this you can take the decision.

Sharepoint Deprecated Feature for SP 2016

With the new release of SP 2016 Microsoft has decided to deprecate few features which are available in the previous version of Sharepoint.

Some of the features which are deprecated were really useful as point of dev perspective.

1) Sharepoint Foundation:

Which is free version of the previous SharePoint that included most of the core functionality and architecture provided by the commercial editions of SharePoint. Not sure about the reason behind may be microsoft looking for more licenced version of the sharepoint which generates the revenue for them :)

2) Stand Alone feature(Sharepoint setup)

Its hard to believe this feature has been deprecated, which is very useful for developer to set up standalone environment in hyper-v/vm. Stand alone feature install and configure every thing for us.

But this has been available in another format as MinRole feature during installation and choose one of the available install options. Using min role we need to install SQL Server and need to run the SharePoint Server 2016 farm configuration wizard.

3) Excel Service

Excel Services functionality is now part of Excel Online in Office Online Server, so this feature has been deprecated but if we migrate the from sharepoint 2013 to 2016 and to retain excel services we need to have Office Online Server with Excel Online

4) ForeFront Identity manager

FIM to synchronize between Active Directory and SharePoint. FIM got replaced with MIM(Microsoft identity Manager) tool to synchronize the data, by default process is Active Directory import. 

5) Sharepoint BI capabilities - Details copied from MSDN

If you want to use Microsoft SQL Server Power Pivot for SharePoint or Microsoft Power View for SharePoint for BI solutions with SharePoint Server 2016 you must install the Power Pivot or Power View add-ins for SQL Server 2016 RTM. The SQL Server 2014 (SP1) Power Pivot for SharePoint and Power View for SharePoint add-ins cannot be deployed or used withSharePoint Server 2016. To deploy these add-ins you need to upgrade to SQL Server 2016 RTM. For more information, see New and improved features in SharePoint Server 2016. The following business intelligence features are available with SharePoint Server 2016 when you download SQL Server 2016 RTM:

6) Tags and Notes

Social feature which was introduced in sharepoint 2010 has been deprecated or microsoft recommends not to use this feature  since it wont be supported from next release onwards

7) STSADM

Those who started working with MOSS are fan of STSADM command until they started using powershell script and its commendlets. Microsoft recommends to use powershell commands to perform daily admin activity.
Support of stsadm will be available for previous version of the sharepoint

Extending Sharepoint WebApplication

What is extending sharepoint webapplicaiton?

Let's take an example to understand this concept.

Assume we have two sharepoint site but the content of both sites are same. Here the question raises do we need to create the same content for both the sites or is there any possibility that sharepoint infrastructure provide us some mechanism to handle this in elegant way.

so here comes the concept of extending the sharepoint web  applicaion:

Expose the same content to two different set of users with different URL's and with different authentication will use  the concept of extending  sharepoint  web applicaiton.

Each  Web  app will assign  to corresponding IIS  web site and when it get extended it will create another web applicaiton and inturn it create new IIS web site as well.
The extended web app will use the same content DB as with  the main web app(here we can call it as parent web applicaiton), since it will create its own  IIS web site it will be having its own Web.config  file.

This concept will be handy when parent site use windows  authentication and extended site uses form based  authentication

Note: By default extended web applicaiton uses the  same authentication as with the parent  site, but we can change  it as per the  requirement.

Uninstall DLL from Gac using Powershell

In some cases we might face issue while deleting the dll from GAC( usually it will say its in use by some process) and it will always annoying, to over come this here is small powershell script which will be handy.

Set-location "Folder path of the DLL"          
[System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")          
$publish = New-Object System.EnterpriseServices.Internal.Publish          
$publish.GacRemove("FolderPath\DLL.dll")          
iisreset