Sharepoint Timer Job Paused

I have created couple of custom timer job and it was working as expected without any issues.

One day when I start the job manually (since I need to pull the data other system and push it to sharepoint list, which was as needed), and it started running without any issues.

But after couple of minutes it started showing paused along with other OOB timer jobs.

This seems to be more interesting why all of sudden all the jobs went to pause state.

Then I started looking all the jobs carefully and I can see there is job called "timer service recycle" which was alone running(couple of instance one in WFE and another one in APP).

Then I started looking into more details about this timer job and what exactly it will do is very well documented in below blog.

Timer Recycle Job

This job will start @6AM in morning and it will run for approximately 11 minutes in the farm.

Once this job finishes automatically all other jobs will start running automatically.

The Execute method of job definition Microsoft.SharePoint.Administration.SPUsageImportJobDefinition threw an Exception

Background:

We are using sharepoint 2013 environment with SharePoint Server with Enterprise Client Access License with SP1 installed(15.0.4433.1507).

Recently there was windows  update which was installed in this Farm(Test instance) and  after that there were couple of issues which started throwing.

In the event viewer I could  see lot of errors with the following error message

The Execute method of job definition Microsoft.SharePoint.Administration.SPUsageImportJobDefinition threw an Exception with the EventID: 6398

Since this is critical error type and Scom started triggering the mails very often.

When I check the error message its saying Access to the path 'C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS' is denied.

This seems to be strange behavior, the account which is running the timer job is part of local admin group which is having full access to the folder path.

Initially I assumed this was due to Diagnostic Logs which is throwing this error but I when I checked it was pointing to D:\Logs folder which I was configured initially and also if that is the case then it should not log the error in periodic basis.

Then I verified the Configure usage and health data collection log path, this was the culprit which was pointing to this error. Later I change the path to D: then all the error stopped logging in event viewer.

But now the question is why it started giving this issue after windows update.

I have raised case with MS to know more about the issues and will update once I get any reply

Sharepoint Search Error 0FF1CE15-013A-0000-0000-000000000000 -Error loading type library/DLL.

I was setting up the sharepoint 2013 server and configured the sharepoint search in one of the app server(1 WFE, 1 APP and SQL server)

Everything was working as  expected, but last Sunday our infrastructure team has done windows update in APP server and SQL server(it seems to be other update as well).

After that there was strange behavior in the farm, like while running the jobs it was throwing access denied error, health usage report etc. So we asked infra team to revert the APP server snapshot to before update installed.

They have reverted the snap shot of the server and once they hand over the server, I checked 
the all the service application which we configured earlier, and Search service application was crashed due to server update happened in the APP server where search is configured and I was getting below error when I open the search service.

The Execute method of job definition Microsoft.SharePoint.Administration.SPServiceInstanceJobDefinition (ID a41c8bc6-6ee7-4039-b171-277686fc0b96) threw an exception. More information is included below.  Unable to cast COM object of type 'Microsoft.Office.Server.Search.Administration.MSSITLB.CGatheringManagerClass' to interface type 'Microsoft.Office.Server.Search.Administration.MSSITLB.IGatherManagerAdmin3'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0FF1CE15-013A-0000-0000-000000000000}' failed due to the following error: Error loading type library/DLL. (Exception from HRESULT: 0x80029C4A (TYPE_E_CANTLOADLIBRARY))."

I thought my search service got screwed up and I need to create it from scratch. But I don't want to do that since everything was working as expected, so I just wanted to deep dive into the issue with ULS Log, Event viewer.

There was one clue was available in the error message i.e, 0FF1CE15-013A-0000-0000-000000000000

When I just saw this id, I assumed its  some thing related to Registry entry, either its having issue while accessing the Registry key or key might be corrupted(I was under assumption that whole registry got corrupted, luckily that was not the case)

Then I opened the Registry 

1) windows + run, type regedit it will open registry application(Make sure you open the registry with admin access else you wont be able to open)

2) Search for this specific key in the registry - Note here flower brackets are very important while searching else search return multiple value

3) This will be located under HKEY_CLASSES_ROOT\Interface folder

4) There will be 3 folder under this as like below

Above pictures shows the path of the key.

5) Here you need to select {0FF1CE15-0001-11B0-0000-000000000000} again curly braces are important and search under the registry folder HKEY_CLASSES_ROOT\TypeLib\

Please follow the path as depicted in the below screen shot.

As  we can see here its having the value "C:\PROGRA~1\MI95D8~1\15.0\Bin\mssrch.dll\1" which we can say its corrupted or its not having the proper location path.

This key value we need to change it to "C:\Program Files\Microsoft Office Servers\15.0\Bin\mssrch.dll\1" and save the entry.

Couple of notes:

1) These changes needs to be done in the server where search is configured, in my case I have done it in app server

2) After registry changes its recommended to restart the machine, please plan for that

3) After this registry changes search service  application will loose all its crawl history, need to run full crawl(basically search will not be in working condition, so we need to run full crawl)

4) In my case I have not created new search service application, existing application worked without any issue.

5) Query processing component will throw error if we search for items, in that time its required to start below services

• SharePoint Server Search
• SharePoint Search Host Controller
• IIS  Reset in all servers(Optional)