Table of Contents

LiteMage Cache Warmup

Magento includes a built-in cronjob to crawl the site and warm up the full page cache. LiteMage also has its own cron job to do this.

To enable LiteMage cache warmup, the following steps should be taken.

Initial Setup

<IfModule Litespeed> 
LiteMage on 
</IfModule>

Recommendation settings are as followed:

Crontab Setup

Setting up the crontab is a fairly simple process that requires shell access. After logging into shell run the following commands:

WARNING! - Do not setup crontab as root user, always specify a user by using -u variable!
Running as root user is a security risk and will cause LiteMage to not work properly!

First we will need to edit the crontab file, replace “magentouser” with the user account that is running magento. 

crontab -u magentouser -e

Second we need to add the following line into the crontab, replace /home/magentouser/public_html/ with the path to your magento install. 

* * * * * /home/magentouser/public_html/cron.sh cron.php

That's it! The LiteMage crawler should now be working.

Check That The Magento cron job Is Working

Issues do happen, here we will go over how to check if cron is executable and to check if the crawler is running.

If you check the permission mask for cron.sh, you will see that it is a non-executable file. 

ls -l cron.sh
-rw-r--r-- 1 magentouser magentouser 746 May  6 17:19 cron.sh

Because of this the cron job will not be executed and must be changed to the following: 

* * * * * /bin/sh /home/magentouser/public_html/cron.sh cron.php

To verify if the cron is running ok, you can check cron log: 

vi /var/log/cron

And you should see something like: 

Oct 25 18:59:01 test CROND[12606]: (magentouser) CMD (/bin/sh /home/magentouser/public_html/cron.sh cron.php)

If everything is okay, LiteMage cache warmup should be executed and can be checked by doing the following: 

cd /magento_installation_directory/var/cache/
find . | grep WARMUP

./mage--6/mage---internal-metadatas---160_LITEMAGE_WARMUP_URLMAP
./mage--6/mage---160_LITEMAGE_WARMUP_URLMAP
./mage--5/mage---160_LITEMAGE_WARMUP_META
./mage--5/mage---internal-metadatas---160_LITEMAGE_WARMUP_META

Note1: var/cache directory is used only if user is using file system based cache. if you use redis, you don't see data there.

Warmup files showing up means everything is okay while not appearing indicates that warmup is not being executed. If this is the case, something is wrong with your LiteMage warmup configuration.

By default, the LiteMage cron job is set to run every 10 minutes (which is set in app/code/community/Litespeed/Litemage/etc/config.xml) to check the warmup.

For more information on configuring and scheduling Magento cron jobs, see Magento User Guide - Setting Up Cron Jobs and Magento User Guide - Cron (Scheduled Tasks).

Changing How Often The LiteMage's Warm Cache cron job Runs

The intervals at which the LiteMage cron job runs can be configured in LiteMages config.xml file located under …/app/code/community/Litespeed/Litemage/etc.

Near the bottom of this file in the you will see the <crontab> tag (directly under <config>). 

<crontab>
        <jobs>
            <litemage_warmup_cache>
                <schedule>
                    <cron_expr>2,12,22,32,42,52 * * * *</cron_expr>
                </schedule>
                <run>
                    <model>litemage/observer_cron::warmCache</model>
                </run>
            </litemage_warmup_cache>
            <litemage_crawl_delta>
                <schedule>
                    <cron_expr>*/5 * * * *</cron_expr>
                </schedule>
                <run>
                    <model>litemage/observer_cron::crawlDelta</model>
                </run>
            </litemage_crawl_delta>
        </jobs>
</crontab>

This section controls the scheduling of LiteMage's cron jobs with the important part being 

<cron_expr>0,10,20,30,40,50 * * * *</cron_expr>

which says to do what is in the following <run> tag, warm the cache in this case, every 10 minutes.

While it is recommended to leave this as is, you can modify it as needed to change the interval at which your site is crawled. So for example, if you wanted to crawl every 5 minutes instead, you could change this line to 

<cron_expr>0,5,10,15,20,25,30,35,40,45,50,55 * * * *</cron_expr>

Note: Create a backup of your config.xml file after making any changes as they will be overwritten when upgrading and need to be re-added.

Configuring LiteMage's Cache Warmup cron job

LiteMage's cache warmup cron job can be configuring through the Magento Admin Panel. This can be done under Cache Warm Up in System→Configuration→LiteMage Configuration.

If you have modified the LiteMage cron job time interval mentioned in the previous section, please ensure that Maximum Run Time (seconds) is set to a lower number of minutes. You can divide your Maximum Run Time setting by 60 to get the value in minutes. For example, a setting of 300 is equal to 300/60 = 5 minutes.

It is possible that LiteMage will not finish crawling your site within the set Max Run Time. This can be caused by a large site or a high server load at run time, where the crawler will stop to prevent it from using up your remaining server resources. In these cases, the crawler should continue where it left off on its next run.

You can check on the crawlers progress at the bottom of the System→Cache Management page in the Magento Admin Panel or click the reset link to restart a crawlers queue without flushing your cache.

To increase the speed at which your site is crawled, try increasing your Maximum Crawler Threads setting.

For more information on the different Cache Warm Up settings, see our Cache Warmup Configuration page.

Defining A Custom Warm Up URL List

Use Cases

  1. You’re front page displays some random products from a certain category, “best sellers” for example. You can set the “Homepage TTL” to a shorter interval like 60 minutes, or set up a custom warm up URL list with a 5 minute interval and a priority of 10 (so every 5 minutes it will be refreshed to show different random items).
  2. You cached “shop_by_brand” pages (these pages will not be auto-purged). You can refresh these pages every 30 minutes to make sure they are accurate.
  3. You have a special configurable product pages with constantly changing inventory. In this case, as the pages are likely very heavy, it may be okay to keep them out-of-date for a while to avoid having to regenerate after flushing.

Benefits

One can define their own list of URLs to be refreshed at specified intervals and priority. During crawling, the LiteMage crawler will hit the backend and regenerate the page, updating the cached copy. The next visitor to the site will then get the newly generated cached copy. Regular visitors will still be served from cache and will not notice that this is taking place. In this way cached copies of certain pages can be updated frequently while still serving all visitors from cache 100% of the time.

Setup

  1. Collect all the related brand URLs and save them into either a single file or multiple files if needed. For example /home/user/public_html/var/litemage/brands.url. Each file listed should contain a list of custom-defined URLs (one per line). These URLs are relative to a store’s base URL (without http://domain/storepath/).
  2. In the Magento Admin Panel, navigate to LiteMage Configuration→Store View and add the created URL lists to Custom Defined URL List Files.

Here, one can specify a list of Custom URL files with varying intervals and priorities. Each line should contain the absolute path to a single file followed by a crawling interval and a priority. The crawling interval must be set to a number in seconds greater than 600. The priority must be set to a number larger than 0, where a lower number means a higher priority.

For example:

One can set the priority for their English store as follows

/home/user/public_html/var/litemage/brands.url 1800 10

And then set the priority of their French store as follows

/home/user/public_html/var/litemage/brands.url 1800 20

This will have the crawler crawl every 30 minutes starting with the English store before moving on to the French store.

Enabling The Debug Log

See: Enabling The Debug Log 

tail -f system.log
grep "crawling url" system.log

will list all crawling urls.

LiteMage Crawler Warm-up Status

In your Magento Admin Panel, at the bottom of the System → Cache Management page, is a section where you can monitor and manage the crawler's status.

Each crawler queue is listed as its own row in the table, sorted ascending based on priority. Included with each crawler listing is:

A crawler queue can be reset by clicking the corresponding “Reset” link in this section. This will remove values for: Custom List Path, Generated, Finished, List Size, and Last Query Time which will be repopulated when the crawler regenerates the list and crawls it. This will also set Current Position and Total Queried back to zero, starting the queue from the beginning the next time the crawler crawls it.

Keeping Public Cache Warm Indefinitely

Using LiteMage's crawler to warm your cache can lead to a huge improvement in performance by regularly re-cacheing pages that might otherwise have expired. By slightly changing your LiteMage configuration, you can set this up so this is always the case.

LiteSpeed Web Server is able to detect the difference between requests coming from the crawler and those from elsewhere allowing it to serve user requests from cache while still letting the crawler hit the backend. This results in your cache being in an always warm state with visitors never hitting the backend.

To do this, change the Default Public Cache TTL setting in the General Settings section of your LiteMage Configuration screen to an amount of time greater than your Warm Up Interval setting in the Cache Warmup section.

For example:

By setting the Default Public Cache TTL to 86400 seconds (24 hours)

and the Warm Up Interval to 43200 seconds (12 hours)

the crawler will update your publicly cached pages before their cache entries have the chance to expire.

Note: This can be done on a per-store basis by changing your Current Configuration Scope.

Note: LiteMage will not crawl your site all at once. In order to prevent the crawler from using all of your server's resources and allow flexibility, the crawler will be run once every 10 minutes (600 seconds). You can set the Maximum Run Time for a crawler run to any value less than 600 seconds, the default duration being 5 minutes (300 seconds). When the crawler reaches this time limit, it stops until it's next run and then resumes where it left off. If your Default Public Cache TTL setting is too low, parts of your site will not remain in cache.

Note: You can restart a crawlers queue without flushing your cache by clicking the reset link at the bottom of the System→Cache Management page in the Magento Admin Panel.

Note: Users can also set their own Custom Defined Url list in case they wish to refresh certain URLs more or less frequently than their crawler queues.

Note: If, after updating a page, a user wants to refresh the cached copy of this page, they can use the following command in a cron job to refresh the cached copy programmatically: 

curl -A "litemage_walker" https://your_url

Refreshing the cache is not the same as invalidating it. When the cached copy is invalidated, the next user to hit the invalidated page will have to wait longer for the page to load as there is no copy to serve from cache. When the cache is refreshed, the cached copy is overwritten with the current version of the page.

The difference between the LiteMage Purge command and the curl command listed above is that the Purge command invalidates the cache for the current page, while the curl command refreshes the cached copy of the given URL.

Warm Up Still Not Working

Magento System Cron issue

If you have triell of the above and LiteMage is still not working properly and showing the Warm-up List details, it is likely that incorrect values in Magento's Cron (Scheduled Tasks) setting is blocking cron jobs.

We suggest returning these settings, located under System > Configuration > Advanced > System > Cron (Scheduled Tasks), to their defaults below to fix the problem. 

  Generate Schedules Every 15
  Schedule Ahead for 20
  Missed if Not Run Within 15
  History Cleanup Every 10
  Success History Lifetime 60
  Failure History Lifetime 600

You can also enable debug log, purge the cache and check the log to see if the crawler is running, expecially during the crawler scheduled time, such as 2-4, 12-14, 22-24, 32-34, 42-44, 52-54.

tail -f var/log/system.log

You should be able to see log for cache purge: 

2017-10-27T20:31:04+00:00 DEBUG (7): LiteMage (4) Reinit Delta queue [time=0.85439600 1509136264] tags=
2017-10-27T20:31:04+00:00 DEBUG (7): LiteMage (3) SetPurgeHeader: * (triggered by purgeAllByAdmin)

then crawler should start running like the following and run for a couple of minutes until all done. 

2017-10-27T20:34:15+00:00 DEBUG (7): LiteMage [litemage_runner:103.16.17.40:52630:76984300] (2) ****** PRECHECK route_action [catalog_category_view] /health.html
2017-10-27T20:34:15+00:00 DEBUG (7): LiteMage [litemage_runner:103.16.17.40:52626:80860200] (2) ****** PRECHECK route_action [catalog_category_view] /special-deals.html
2017-10-27T20:34:15+00:00 DEBUG (7): LiteMage [litemage_runner:103.16.17.40:52628:82403400] (2) ****** PRECHECK route_action [catalog_category_view] /fitness.html
2017-10-27T20:34:15+00:00 DEBUG (7): LiteMage [litemage_runner:103.16.17.40:52624:88625900] (2) ****** PRECHECK route_action [cms_index_index] /
2017-10-27T20:34:16+00:00 DEBUG (7): LiteMage [litemage_runner:103.16.17.40:52632:07812400] (2) ****** PRECHECK route_action [catalog_category_view] /beauty.html
2017-10-27T20:34:25+00:00 DEBUG (7): LiteMage [litemage_runner:103.16.17.40:52624:88625900] (3) Set Header X-Litespeed-Tag: G.9,S.1
2017-10-27T20:34:25+00:00 DEBUG (7): LiteMage [litemage_runner:103.16.17.40:52624:88625900] (3) Set Header X-Litespeed-Cache-Control: public,max-age=43200,es

… … …

If nothing shows in the log, then there is still something wrong either on configuration or system cron job.

You should also check cron process 

ps -ef | grep cron.php

You should be able to see something similar like the following, if not, most likely your system cron has some problem. 

dealwel+ 4489 1 10 04:23 pts/0 00:00:17 /usr/local/lsws/fcgi-bin/lsphp5 /home/dealwellne/domains/example.com/public_html/cron.php

cron.sh script issue

We did see some case that system cron seems to be showing in the cron log in /var/log/cron but actually there is no such progress running. It did start to run when we change to documentroot and manually run the cron job.

vi /home/dealwellne/domains/example.com/public_html/ 

 
 cd /home/dealwellne/domains/example.com/public_html/
 sudo -u dealwellne ./cron.sh

then we modified the cron.sh script to reflect the above and everything runs ok. 

vi cron.sh
#!/bin/sh
# location of the php binary
if [ ! "$1" = "" ] ; then
CRONSCRIPT=$1
else
CRONSCRIPT=cron.php
fi
#PHP_BIN=`which php`
PHP_BIN=/usr/local/lsws/fcgi-bin/lsphp5
# absolute path to magento installation
INSTALLDIR=`/bin/echo $0 | sed 's/cron\.sh//g'`
#       prepend the intallation path if not given an absolute path
if [ "$INSTALLDIR" != "" -a "`expr index $CRONSCRIPT /`" != "1" ];then
  if ! ps auxwww | grep "$INSTALLDIR""$CRONSCRIPT" | grep -v grep 1>/dev/null 2>/dev/null ; then
      echo "hello world"
      echo "$PHP_BIN $INSTALLDIR$CRONSCRIPT"
      cd /home/dealwellne/domains/example.com/public_html/
      $PHP_BIN ./cron.php &
#       $PHP_BIN "$INSTALLDIR""$CRONSCRIPT" &
  fi
else
  if  ! ps auxwww | grep " $CRONSCRIPT" | grep -v grep | grep -v cron.sh 1>/dev/null 2>/dev/null ; then
      $PHP_BIN $CRONSCRIPT &
  fi
fi

Magento AOE Scheduler doesn't mark completion of ''litemage_warmup_cache''

Magento AOE Scheduler extension sits on top of Magento's default cron functionality allowing the users to manage the jobs and visualize the timeline.

LiteMage works with AOE scheduler most of the time. However, occasionally, Magento AOE scheduler sometimes believers the LiteMage warmup process is always running while it has stopped by the max execution time hence AOE scheduler doesn't start next litemage_warmup_cache at all.

If you navigate to Magento Admin > Scheduler > Scheduler Timeline, you might see litemage_warmup_cache highlighted either in red or orange. It means the cron is hanging and is not running propery.

Navigate to Magento Admin > System > Scheduler > Job Configuration, select litemage_warmup_cache then in the right top corner Actions, select Run now, then press the Submit button.

Check the LiteMage cron log, like so: 

tail -f /home/jsmith/public_html/var/log/litemage_crawler.log
2019-05-29T10:14:21+00:00 DEBUG (7): LiteMage [cron:jsmith:1559124843]  Generate url map for store1 url count=5320
2019-05-29T10:14:21+00:00 DEBUG (7): LiteMage [cron:jsmith:1559124843]  litemage_runner crawling  store1 urls (cur_pos:0) with cookie 
litemage_cron=store1; Array
LiteMage [cron:jsmith:1559124843]   (
LiteMage [cron:jsmith:1559124843]       [0] => https://www.example.com/
LiteMage [cron:jsmith:1559124843]       [1] => https://www.example.com/business-office-writing/
LiteMage [cron:jsmith:1559124843]       [2] => https://www.example.com/business-office-writing/conference-business/business-card-holders/
LiteMage [cron:jsmith:1559124843]       [3] => https://www.example.com/business-office-writing/paper/promotional-calendars-diaries/
LiteMage [cron:jsmith:1559124843]       [4] => https://www.example.com/business-office-writing/desk/promotional-calculators/
LiteMage [cron:jsmith:1559124843]       [5] => https://www.example.com/business-office-writing/paper/promotional-notebooks/
LiteMage [cron:jsmith:1559124843]   )
LiteMage [cron:jsmith:1559124843]
2019-05-29T10:14:32+00:00 DEBUG (7): LiteMage [cron:jsmith:1559124843]  litemage_runner crawling  store1 urls (cur_pos:6) with cookie 
litemage_cron=store1; Array
LiteMage [cron:jsmith:1559124843]   (
LiteMage [cron:jsmith:1559124843]       [0] => https://www.example.com/bags-and-travel/
LiteMage [cron:jsmith:1559124843]       [1] => https://www.example.com/clothing-accessories/
LiteMage [cron:jsmith:1559124843]       [2] => https://www.example.com/usb-technology/
LiteMage [cron:jsmith:1559124843]       [3] => https://www.example.com/food-and-drink/
LiteMage [cron:jsmith:1559124843]       [4] => https://www.example.com/health-personal-care/
LiteMage [cron:jsmith:1559124843]       [5] => https://www.example.com/bags-and-travel/travel/backpacks/
LiteMage [cron:jsmith:1559124843]   )

These results mean that the warmup did begin to run, so we know it works when started manually. A Run now action won't fix the Magento system scheduler problem, however, and the initiated cron will be stopped due to exceeding the defined Maximum Run Time in the configuration.

Take a look at litemage_crawler.log and you will see something like the following: 

2019-06-19T16:17:05+00:00 DEBUG (7): LiteMage [cron:brandin1:1560960721]  Stopped due to exceeding defined Maximum Run Time. cron meta end = Array

This is normal, since the cron was set to stop at 5 minutes before running again at the next scheduled time. If it had stopped there completely without further running, then there would be some Magento system scheduler issue which would need to be fixed.

By taking a look at your crontab (assuming your userid is jsmith) we see that you have too many of the same crons set up which may conflict with each other: 

crontab -u jsmith -e

MAILTO=""
SHELL="/usr/local/cpanel/bin/jailshell"
* * * * * /usr/local/bin/php /home/jsmith/public_html/cron.php
SHELL="/usr/local/cpanel/bin/jailshell"
* * * * * /usr/local/bin/php /home/jsmith/public_html/cron.sh
SHELL="/usr/local/cpanel/bin/jailshell"
* * * * * /bin/bash /home/jsmith/public_html/scheduler_cron.sh --mode always
SHELL="/usr/local/cpanel/bin/jailshell"
* * * * * /bin/bash /home/jsmith/public_html/scheduler_cron.sh --mode default
SHELL="/usr/local/cpanel/bin/jailshell"
* * * * * /bin/bash /home/jsmith/public_html/cron.sh
SHELL="/usr/local/cpanel/bin/jailshell"
* * * * * /bin/sh /home/jsmith/public_html/cron.sh
* * * * * /bin/sh /home/jsmith/public_html/cron.sh cron.php

NOTE: You may need SHELL=“/usr/local/cpanel/bin/jailshell” on cPanel for jailshell.

The latest 1.9.x.x version of Magento may prefer to run /home/jsmith/public_html/scheduler_cron.sh instead of /home/jsmith/public_html/cron.sh. You can find this in Magento Admin > System > Scheduler > Insctuctions: 

Run scheduler_cron.sh instead cron.sh (recommended)
Run every minute (recommended)
Edit your crontab:
sudo crontab -u jsmith -e
Add following configuration:
* * * * * /bin/bash /home/jsmith/public_html/scheduler_cron.sh --mode always
* * * * * /bin/bash /home/jsmith/public_html/scheduler_cron.sh --mode default

Remove all duplicated cron commands and only keep the above two configurations.

800

The above-scheduled tasks view show the AOE scheduler's problem. litemage_warmup_cache was scheduled for 17:02 and run for 5 mins. However, AOE scheduler did not mark the task as “completed” at 17:07 and treats it running all the time. Then 17:12, 17:22, 12:32 etc scheduled litemage_warmup_cache never run due to the incorrect status issue for 17:02 's job. It may need your Magento developer or AOE developer to fix such issue to make litemage_warmup_cache run without any problem until the completion.

Warm Up for logged in users

Normally logged in users and non-logged in users will share the same cache copies. However, when you enable Separate Cache Copy per Customer Group in LiteMage configuration, LiteMage will have different cache copies.

The crawler warmup guide above only shows how to warm up the cache for non-logged in users. However, if you don't enable the crawler for logged-in users, the logged-in page may run slow during the first visit when there's a cache miss.

Enabling Cache for Logged-in Users

You should use the Crawl on Behalf of Logged-in Users setting in LiteMage's Cache Warm Up configuration. Enter user IDs from representative groups, separating them by space or comma. There is no need to place all user IDs. Any single user ID from that user group will do, since every user in that group will share the same cache copy. If you have enabled multi-customer groups, one user id from each group will do.

For the example from the above, with only distinguish between logged-in vs logged-out user selected, all logged-in users will be in the same group, and any logged-in user ID will do the job.

You should be able to get a user ID from the Magento admin customer menu. You can search for any particular email to find an ID.

Once you have the IDs, you can place them in Crawl on Behalf of Logged-in Users.

Save the configuration will make the change effective.

LiteMage Cache Warm Up configuration Reference

Please refer this for LiteMage Cache Warm Up configuration Reference