Sharepoint Configuration Cache Issue

Sharepoint Configuration Cache Overview:

There are many types of cache available in sharepoint environment like Blob cache, Page output cache, Object Cache, Anonymous search results cache, but in this article we are discussing about sharepoint configuration cache which is little bit different than what its mentioned earlier. 

So what is configuration cache. 

Basically all the sharepoint farm configuration details will be cached in system drive in %system drive%\\ProgramData\Microsoft\SharePoint\Config

If we open this folder we can see bunch of folders with GUID as the name, but we wont understand what its corresponds to. The main file which will be used for caching the config data is cache.ini(the statement is false, since it doesn't contain anything except 5/6 digit number). If the environment is having many Guid folder then we can search for cache.ini file

Next question, who will create this files?

There is sharepoint timer job called "Config Refresh" which will create this xml file and update the cache.ini file. So it means first point to trouble shoot the issue(if we have any) is the account which is running sharepoint timer service have enough access permission for R/W operation in that folder. Just cross verify account is having enough permission.

What is the significance of  this XML files.

These XML files points to sharepoint configuration entry which are locally stored in the server. Means whenever there is request for accessing the sharepoint config DB, it will be served from local file system in order to avoid the round trip to SQL server, in order to make response fast.

Ex: If we want to deploy the wsp in the farm using powershell script, these cache files should be in sync with sharepoint config DB else it will start throwing error(some weird errors like operation is not valid due to current state of the object)

Other type of issue is Timer job get paused, and it wont come back to active unless its hit again(start again)

Like this we will face so many issue if config cache folder is not sync. The timer job "Config Refresh" will always make attempt to make it up to date with DB, but there will be few occasions like windows updates/ patch update/ cumulative update might break this relation, after that we will start seeing weird behavior in the environment.

In order to fix this issue there are options available 

The below steps needs to be performed in all the servers in the farm, here I am providing the steps for stand alone system

1) Manually clean up the things

a)    Stop the SharePoint Timer service
.      Here we can goto services and stop the sharepoint timer service else use command       prompt to stop the service(net stop SPTimerv4), reason behind stopping timer service is to stop running "Config Refresh" job to execute else we wont be able to perform the below steps, it will give files are in use.
b)     Navigate to the directory: C:\ProgramData\Microsoft\SharePoint\Config\GUID.
c)     Delete only the XML files in the folder, after this step we should be able to see only cache.ini file in the folder
d)     Open the cache.ini with Notepad(its advisable to open it with admin privilege) and reset the number to 1. Save and close the file.
e)      Start the SharePoint Timer service either manually or through command prompt.
in couple of minutes XML files start creating in this location which indicates config refresh job started running and cache is in sync with config db

We can perform the same activity in other server if the environment is configured  for farm- This activity needs to be performed in all the servers including WFE also)

2) Use Powershell script

In Microsoft TechNet gallery we can find script for cleaning up the cache

Here

By above steps we should be make our environment in sync, but there are few odd sceanrios where accidently we deleted cache.ini files or if the XML files not starting generating again after 5/10 minutes

In that scenario we need to clean all the folder in the %system drive%\\ProgramData\Microsoft\SharePoint\Config (again before doing this we need to stop the timer service)

Then follow the below steps

a) Open Regedit(Registry editor) from start menu
b) Navigate to the below path
 HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14\Secure\ConfigDB
Get the GUID value from the ID attribute
c) Create folder with that GUID(copied) in the %system drive%\\ProgramData\Microsoft\SharePoint\Config folder and create new file from your favorite editor and call it as cache.ini and enter the number 1 and save, close
d) Again start the timer job and after couple of minutes xml files should  start appearing in the folder and if we open the cache.ini file the value will be 5/6 digit number.

Hopefully we should get some basic idea about what is config cache and how to clean up the things if it is not in sync and also its good  idea to clean up the things once in a month to avoid abrupt changes/scenarios

Time out issue in MM service

Some  times we usually get time out exception while accessing/updating term store(Managed  metadata store).

There are several reasons behind this, if you are getting this error on random basis, then it might be due to following reason:

Sharepoint farm is spread  across different servers like WFE, APP servers or we can call it as multi tier architecture. In this type of scenario usually network team recommend to keep WFE in different DMZ(public network) and APP servers in private network, and if the code is executing in WFE for any reason(it might be due to some design constraint or architecture of farm is like that) in that there will be delay in while accessing service( prod server should be capable of handling this issue) may be network failure or any other reason, it will throw the below exception:

System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:30. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

In order to fix these type of issue we need to increase the time out value in below places where metadata service is provisioned:

C:\Program Files\Microsoft Office Servers\15.0\WebClients\Metadata

Open client config file and increase the value from 30 seconds to 1 minute or may be 3 minutes

C:\Program Files\Microsoft Office Servers\15.0\WebServices\Metadata

Open web config file and  increase the value from 30 seconds to 1 minute or may be 3 minutes

Now do an IISRESET in the server where you have done these changes

Going forward we should not get this error.

Cause:
1) Network issue
2) When we perform an bulk operation in term store and it wont be handle such large object in specified time duration
3) Password change(this is very very rare scenario usually admin team keep password of these service account not expired)

There is detailed article from MSDN about this but in my case I have solved the problem with above steps/checks

https://docs.microsoft.com/en-us/previous-versions/office/sharepoint-server-2010/gg681889(v=office.14)#timeoutexception

Localization of Sharepoint Display template

There was requirement in our project where we need to localize the display template. Basically we have 5 different flavors of localization implemented  in our project.

In order to achieve this we have  come up with localizing the resource file with javascript implementation, which is quite interesting and new in sharepoint implementation perspective.

Below are the steps we need to follow to make sure localization can be implemented in display templates

 a) Come up with all localization text  and corresponding key value pair as like normal resource file.
 b) Create javascript file with your favourite editor such  as notepad, notepad++ etc with the below syntax


$registerResourceDictionary("en-us",
{"PageTitle": "Page Title",
"ProductName": "Product Name",
"ProductDescription": "Product Description"});

 c) Here we can all the resource file key value pair which needs to be displayed in the display template

 d) Save the file as ResourceFile.js
 e) Please make sure about the locale language which needs to be present in the 1st  line.
 f) Upload this file in style library and add the reference of this file in the display template as below  reference

g) The resource file needs to be present in all locale site collection style library with appropriate language else it will start throwing error.

h) Once all the above steps completed, then we need to access the resource key like this.

    var pageTitle = $resource("PageTitle");

i) We can use this in the variable in any of the place in the display template

Enabling Developer Dashboard in sharepoint 2013

Developer dash board is one the great feature introduced by Microsoft from sharepoint 2010 version, it got extended in sharepoint 2013 with little bit changes.

Now in this post I will explain how to enable Developer Dashboard in sharepoint 2013 environment.

Below are steps to enable:

1) Make sure Usage and Health data collection service is enabled or not, if its enabled then you can continue with step 3 else need to follow step 2

2) Enable Usage and Health data collection service please refer the msdn link

https://technet.microsoft.com/en-us/library/ee663480.aspx

3) There is no UI steps available to enable Developer Dashboard in sharepoint, so we need to depend only on PowerShell scripts:

4) Open sharepoint 2013 management shell as admin user

5) Run the following code snippet to enable the developer dashboard

$content = ([Microsoft.SharePoint.Administration.SPWebService]::ContentService)
$dashboardsetting =$content.DeveloperDashboardSettings
$dashboardsetting.DisplayLevel = [Microsoft.SharePoint.Administration.SPDeveloperDashboardLevel]::On
$dashboardsetting.Update() 

6) This will enable the developer dash board on the sharepoint site

7) The above steps work only for sites which are opened with intranet site with authenticated users, it doesn't work for anonymous web site. By default developer dashboard will be enabled for authenticated users  with RequiredPermissions set to AddAndCustomizePages

Note: After executing the step 5 if we enter $dashboardsetting in powershell console it will display the all the configuration done with Developer dashboard

8) To enable Developer dashboard in anonymous site we need to run the below script to make it available

$contentService = [Microsoft.SharePoint.Administration.SPWebService]::ContentService
$dashboardSettings = $contentService.DeveloperDashboardSettings
$dashboardSettings.RequiredPermissions = [Microsoft.SharePoint.SPBasePermissions]::EmptyMask
$dashboardSettings.Update()

Note:
a) Make sure Developer Dashboard html tags are available in the master page(in case if the master page is customized)
b) Usage and Health data collection service is up and running
c) I have verified developer dashboard for https site also with anonymous users(internet facing site) and its working without any issues.

HTTP to HTTPS redirection

In my current assignment I have created new web application under port 443(with host header site collection) and  enabled both anonymous and NTLM authentication.
And in IIS I have edited the bindings used certificate to make it https site.

Added DNS entry and  it was working as expected.

Ex: If the user browse the site https://test.abc.com its opening the site, but if the user enter

http://test.abc.com then it was not redirecting to my https site and also it was giving 404 error.

From url perspective what ever it was returning that was correct, since  there was no site hosted under port 80 and it was giving 404 error.

Later I thought I will use the IIS URL Redirect module which is available and setup the same in the environment, but it was not working as expected.

In order to fix this later I decided I will add new AAM in the web application under default zone with http url

ex: 

Here I have created AAM with default zone(even though there is default zone already present) with http://test.abc.com

Now bind that host entry in IIS web application like as below

Now we have all the entry for https and http (ignore the other 2 which I have created for my project purpose).

If the user now tries to browse the site with http it will redirect to https

Note:

My IIS application was created under port 443

Its binded with SSL and its extended web application

Sharepoint Timer Job Paused

I have created couple of custom timer job and it was working as expected without any issues.

One day when I start the job manually (since I need to pull the data other system and push it to sharepoint list, which was as needed), and it started running without any issues.

But after couple of minutes it started showing paused along with other OOB timer jobs.

This seems to be more interesting why all of sudden all the jobs went to pause state.

Then I started looking all the jobs carefully and I can see there is job called "timer service recycle" which was alone running(couple of instance one in WFE and another one in APP).

Then I started looking into more details about this timer job and what exactly it will do is very well documented in below blog.

Timer Recycle Job

This job will start @6AM in morning and it will run for approximately 11 minutes in the farm.

Once this job finishes automatically all other jobs will start running automatically.

The Execute method of job definition Microsoft.SharePoint.Administration.SPUsageImportJobDefinition threw an Exception

Background:

We are using sharepoint 2013 environment with SharePoint Server with Enterprise Client Access License with SP1 installed(15.0.4433.1507).

Recently there was windows  update which was installed in this Farm(Test instance) and  after that there were couple of issues which started throwing.

In the event viewer I could  see lot of errors with the following error message

The Execute method of job definition Microsoft.SharePoint.Administration.SPUsageImportJobDefinition threw an Exception with the EventID: 6398

Since this is critical error type and Scom started triggering the mails very often.

When I check the error message its saying Access to the path 'C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\LOGS' is denied.

This seems to be strange behavior, the account which is running the timer job is part of local admin group which is having full access to the folder path.

Initially I assumed this was due to Diagnostic Logs which is throwing this error but I when I checked it was pointing to D:\Logs folder which I was configured initially and also if that is the case then it should not log the error in periodic basis.

Then I verified the Configure usage and health data collection log path, this was the culprit which was pointing to this error. Later I change the path to D: then all the error stopped logging in event viewer.

But now the question is why it started giving this issue after windows update.

I have raised case with MS to know more about the issues and will update once I get any reply

Sharepoint Search Error 0FF1CE15-013A-0000-0000-000000000000 -Error loading type library/DLL.

I was setting up the sharepoint 2013 server and configured the sharepoint search in one of the app server(1 WFE, 1 APP and SQL server)

Everything was working as  expected, but last Sunday our infrastructure team has done windows update in APP server and SQL server(it seems to be other update as well).

After that there was strange behavior in the farm, like while running the jobs it was throwing access denied error, health usage report etc. So we asked infra team to revert the APP server snapshot to before update installed.

They have reverted the snap shot of the server and once they hand over the server, I checked 
the all the service application which we configured earlier, and Search service application was crashed due to server update happened in the APP server where search is configured and I was getting below error when I open the search service.

The Execute method of job definition Microsoft.SharePoint.Administration.SPServiceInstanceJobDefinition (ID a41c8bc6-6ee7-4039-b171-277686fc0b96) threw an exception. More information is included below.  Unable to cast COM object of type 'Microsoft.Office.Server.Search.Administration.MSSITLB.CGatheringManagerClass' to interface type 'Microsoft.Office.Server.Search.Administration.MSSITLB.IGatherManagerAdmin3'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0FF1CE15-013A-0000-0000-000000000000}' failed due to the following error: Error loading type library/DLL. (Exception from HRESULT: 0x80029C4A (TYPE_E_CANTLOADLIBRARY))."

I thought my search service got screwed up and I need to create it from scratch. But I don't want to do that since everything was working as expected, so I just wanted to deep dive into the issue with ULS Log, Event viewer.

There was one clue was available in the error message i.e, 0FF1CE15-013A-0000-0000-000000000000

When I just saw this id, I assumed its  some thing related to Registry entry, either its having issue while accessing the Registry key or key might be corrupted(I was under assumption that whole registry got corrupted, luckily that was not the case)

Then I opened the Registry 

1) windows + run, type regedit it will open registry application(Make sure you open the registry with admin access else you wont be able to open)

2) Search for this specific key in the registry - Note here flower brackets are very important while searching else search return multiple value

3) This will be located under HKEY_CLASSES_ROOT\Interface folder

4) There will be 3 folder under this as like below

Above pictures shows the path of the key.

5) Here you need to select {0FF1CE15-0001-11B0-0000-000000000000} again curly braces are important and search under the registry folder HKEY_CLASSES_ROOT\TypeLib\

Please follow the path as depicted in the below screen shot.

As  we can see here its having the value "C:\PROGRA~1\MI95D8~1\15.0\Bin\mssrch.dll\1" which we can say its corrupted or its not having the proper location path.

This key value we need to change it to "C:\Program Files\Microsoft Office Servers\15.0\Bin\mssrch.dll\1" and save the entry.

Couple of notes:

1) These changes needs to be done in the server where search is configured, in my case I have done it in app server

2) After registry changes its recommended to restart the machine, please plan for that

3) After this registry changes search service  application will loose all its crawl history, need to run full crawl(basically search will not be in working condition, so we need to run full crawl)

4) In my case I have not created new search service application, existing application worked without any issue.

5) Query processing component will throw error if we search for items, in that time its required to start below services

• SharePoint Server Search
• SharePoint Search Host Controller
• IIS  Reset in all servers(Optional)