Advanced crawler for recache. some ideas and code

AndreyPopov · Aug 25, 2021

Lee said:

anywhere in file.
for example after or before

PHP:

protected function CountNumberOfPages($filter_data) {
.......
}

these only needed functions.
recache() function that must be replaced not yet posted.

AndreyPopov · Aug 25, 2021

functions to build urls in one file (all above)

AndreyPopov · Aug 25, 2021

NEXT step - replace whole function recache()

in original begin at line 721

Code:

    public function recache()
    {
    ...........
    }

end at line 861

by code from file

AndreyPopov · Aug 25, 2021

NEXT

- change line

PHP:

private function crawlUrls($urls, $cli = false)

to

PHP:

private function crawlUrls($urls, $cli=false, $showpathcount=false, $pathcount=1, $totalurls = 0 )

- after line

PHP:

$url = str_replace('&amp;', '&', $url);

add

PHP:

$url = str_replace('&amp%3B', '&', $url);
$url = str_replace('?amp%3B', '?', $url);
$url = str_replace('page%3D', '&page=', $url);

- replace lines

PHP:

            if ($cli) {
                echo $current . '/' . $count . ' ' . $url . ' : ' . $httpcode . PHP_EOL;
            } else {
                echo $current . '/' . $count . ' ' . $url . ' : ' . $httpcode . '<br/>' . PHP_EOL;
            }

with

PHP:

                if ( $showpathcount ) {
                    echo $current . '/' . $count . '/' . $pathcount . '/' . $totalurls . ' ' . $url . ' : ' . $httpcode . ($cli ? '' : '<br>') . PHP_EOL;
                } else {
                    echo $current . '/' . $count . ' ' . $url . ' : ' . $httpcode . ($cli ? '' : '<br>') . PHP_EOL;
                }

AndreyPopov · Aug 25, 2021

NEXT

- after lines

PHP:

        $includeUrls = isset($this->lscache->setting['module_lscache_include_urls']) ? explode(PHP_EOL, $this->lscache->setting['module_lscache_include_urls']) : null;
        $this->lscache->includeUrls = $includeUrls;

add

PHP:

        // additional sorts and filters to recache_options
        $includeSorts = isset($this->lscache->setting['module_lscache_include_sorts']) ? preg_split( '/\n|\r\n?/', $this->lscache->setting['module_lscache_include_sorts']) : null;
        $this->lscache->includeSorts=$includeSorts;
        $includeFilters = isset($this->lscache->setting['module_lscache_include_filters']) ? preg_split( '/\n|\r\n?/', $this->lscache->setting['module_lscache_include_filters']) : null;
        $this->lscache->includeFilters=$includeFilters;

AndreyPopov · Aug 25, 2021

to be continued .....

Lee · Aug 25, 2021

Makes more sense now!

Thank you...

AndreyPopov · Aug 26, 2021

NEXT

appear in GUI

to file admin/language/en-gb/extension/module/lscache.php
add

PHP:

// Include Sorts/Filters
$_['entry_include_sorts']     = 'Include Sorts/Filters';
$_['entry_include_filters']     = 'Include Journal Specific';
$_['entry_module_lscache_product_list_recache_total']     = 'Total Product URLs';
$_['entry_module_lscache_product_list_last_recached']     = 'Already Recached Product URLs';
$_['entry_module_lscache_product_list_recache_status']     = 'Product URLs List Status';
$_['hint_include_sorts'] = "Please input one kind of sorts or filter per line. \neg: \nsort=p.date_added&order=DESC \neg: \nfm=11";
$_['hint_include_filters'] = "Please input one Journal specific product url/popup per line. \neg: \npopup=options&product_quantity=1&";
$_['help_include_sorts']     = 'Additional sorts/filters to recache';
$_['help_include_filters']     = 'Additional Journal specific product url/popup to recache';

// What Recache
$_['entry_recache_option_what']     = 'Choose what to recache';
$_['entry_recache_option_what_button']     = 'Recache';
$_['text_recache_what_default']     = 'recache whole site';
$_['text_recache_what_products']     = 'recache only products';
$_['text_recache_what_category']     = 'recache only categories';
$_['text_recache_what_manufacturer']     = 'recache only manufacturers';
$_['text_recache_what_catalog']     = 'recache only catalog';
$_['text_recache_what_category_products']     = 'recache categories and products';
$_['text_recache_what_manufacturer_category_products']     = 'recache manufacturers, categories and products';
$_['text_recache_what_manufacturer_products']     = 'recache manufacturers and products';
$_['text_recache_what_catalog_products']     = 'recache catalog and products';
$_['text_recache_what_catalog_category_products']     = 'recache catalog, categories and products';
$_['text_recache_what_catalog_manufacturer']     = 'recache catalog and manufacturers';
$_['text_recache_what_catalog_manufacturer_products']     = 'recache catalog, manufacturers and products';
$_['button_recacheWhat'] = 'Rebuild Part of LiteSpeed Cache by your choose';

AndreyPopov · Aug 26, 2021

NEXT

to file admin/controller/extension/module/lscache.php

after line

PHP:

$data["button_recacheAll"] .= $data["text_curl_not_support"];

add

PHP:

$data["button_recacheWhat"] .= $data["text_curl_not_support"];

after line

PHP:

$recacheLink = $siteUrl->link('extension/module/lscache/recache', 'user_token=' . $this->session->data['user_token'],true);

add

PHP:

        $this->load->model('extension/module/lscache');
        $settingWhat = $this->model_extension_module_lscache->getItems();
        $recacheLinkWhat = $siteUrl->link('extension/module/lscache/recache', 'what=' . $settingWhat['module_lscache_recache_option_what'] . '&user_token=' . $this->session->data['user_token'],true);

after lines

PHP:

            if(!isset($oldSetting["module_lscache_vary_safari"])){
                $oldSetting["module_lscache_vary_safari"] = '0';
            }

add

PHP:

            // add variables for what recache
            if(!isset($oldSetting["module_lscache_recache_option_what"])){
                $oldSetting["module_lscache_recache_option_what"] = '1111';
            }

after line

PHP:

$data['recacheAll'] = $this->isCurl()? $recacheLink : '#';

add

PHP:

$data['recacheWhat'] = $this->isCurl()? $recacheLinkWhat : '#';

to file admin/model/extension/module/lscache.php

after line

PHP:

$this->db->query(" insert into " . DB_PREFIX . "setting (store_id, code, `key`, value, serialized) values ('0', 'module_lscache', 'module_lscache_vary_safari', '0', '0')") ;

add

PHP:

        // lscache crawler what recache variabels
        $this->db->query(" insert into " . DB_PREFIX . "setting (store_id, code, `key`, value, serialized) values ('0', 'module_lscache', 'module_lscache_recache_option_what', '1111', '0')") ;

AndreyPopov · Aug 26, 2021

NEXT

in file admin/view/template/extension/module/lscache.twig

after lines

PHP:

                <div class="form-group">
                    <label class="col-sm-2 control-label" for="module_lscache_include_urls"><span data-toggle="tooltip" title="{{ help_include_urls }}">{{ entry_include_urls }}</span></label>
                  <div class="col-sm-10">
                    <textarea  name="module_lscache_include_urls" placeholder="{{ hint_include_urls }}" class="form-control" rows="3">{{ module_lscache_include_urls }}</textarea>
                  </div>
                </div>

add

PHP:

                <!-- Include sorts to recache -->
                <div class="form-group">
                    <label class="col-sm-2 control-label" for="module_lscache_include_sorts"><span data-toggle="tooltip" title="{{ help_include_sorts }}">{{ entry_include_sorts }}</span></label>
                  <div class="col-sm-10">
                    <textarea  name="module_lscache_include_sorts" placeholder="{{ hint_include_sorts }}" class="form-control" rows="3">{{ module_lscache_include_sorts }}</textarea>
                  </div>
                </div>


                <!-- Include filter to recache -->
                <div class="form-group">
                    <label class="col-sm-2 control-label" for="module_lscache_include_filetrs"><span data-toggle="tooltip" title="{{ help_include_filters }}">{{ entry_include_filters }}</span></label>
                  <div class="col-sm-10">
                    <textarea  name="module_lscache_include_filters" placeholder="{{ hint_include_filters }}" class="form-control" rows="3">{{ module_lscache_include_filters }}</textarea>
                  </div>
                </div>

after lines

PHP:

                <div class="form-group">
                  <label class="col-sm-2 control-label" for="module_lscache_purge_url">
                      <button type="submit" name='purgeURL' form="form-lscache-module" data-toggle="tooltip" class="btn btn-warning"><i class="fa fa-trash"> &nbsp; {{ entry_purge_urls }}</i></button>
                  </label>
                  <div class="col-sm-10">
                    <textarea  name="lscache_purge_url" placeholder="{{ hint_purge_urls }}" class="form-control" rows="3">{{ lscache_purge_urls }}</textarea>
                  </div>
                </div>

add

PHP:

                <!-- Choose what to recache -->
                <div class="form-group">
                  <label class="col-sm-2 control-label" for="module_lscache_recache_option_what">{{ entry_recache_option_what }}
                    <a href="{{ recacheWhat }}" data-toggle="tooltip" title="{{ button_recacheWhat }}" class="btn btn-success"><i class="fa fa-flash"> &nbsp; </i></a>
                  </label>
                  
                  <div class="col-sm-10">
                    <select name="module_lscache_recache_option_what" class="form-control">
                      <option value="1111" {{ selectDisable.check(module_lscache_recache_option_what, '1111') }}>{{ text_recache_what_default }}</option>
                      <option value="1" {{ selectDisable.check(module_lscache_recache_option_what, '1') }}>{{ text_recache_what_products }}</option>
                      <option value="10" {{ selectDisable.check(module_lscache_recache_option_what, '10') }}>{{ text_recache_what_category }}</option>
                      <option value="100" {{ selectDisable.check(module_lscache_recache_option_what, '100') }}>{{ text_recache_what_manufacturer }}</option>
                      <option value="1000" {{ selectDisable.check(module_lscache_recache_option_what, '1000') }}>{{ text_recache_what_catalog }}</option>
                      <option value="11" {{ selectDisable.check(module_lscache_recache_option_what, '11') }}>{{ text_recache_what_category_products }}</option>
                      <option value="111" {{ selectDisable.check(module_lscache_recache_option_what, '111') }}>{{ text_recache_what_manufacturer_category_products }}</option>
                      <option value="1001" {{ selectDisable.check(module_lscache_recache_option_what, '1001') }}>{{ text_recache_what_catalog_products }}</option>                     
                      <option value="1011" {{ selectDisable.check(module_lscache_recache_option_what, '1011') }}>{{ text_recache_what_catalog_category_products }}</option>
                      <option value="1100" {{ selectDisable.check(module_lscache_recache_option_what, '1100') }}>{{ text_recache_what_catalog_manufacturer }}</option>
                      <option value="1101" {{ selectDisable.check(module_lscache_recache_option_what, '1101') }}>{{ text_recache_what_catalog_manufacturer_products }}</option>
                      <option value="101" {{ selectDisable.check(module_lscache_recache_option_what, '101') }}>{{ text_recache_what_manufacturer_products }}</option>
                    </select>
                  </div>
                </div>

AndreyPopov · Aug 26, 2021

some issues and how-to-use - later

P.S. if anybody know how to replace block of code by ocmod search regex then I can make modification module.

Lee · Aug 26, 2021

Good heavens you put a ton of work into this! Bravo.

I will try and find time today to put it all together and see what happens...

AndreyPopov · Aug 26, 2021

after edit twig file not forget clear SASS and Theme cache (main admin screen, right up corner )

fill fields, make your choose and click Save.

ATTENTION!!!!! after each Save you must press twice "Rebuild Part of LiteSpeed Cache" before run recache process.

first click (click and wait) start initial process that check and prepare variables.
second click (click and wait) start process to build of urls list.
and only third (and next clicks) start recache process.

(I do not find what I must to do that after just writing to database I can read these settings ????????)

Lee · Aug 27, 2021

Here is an updated ocmod file, I had some duplicates...

Lee · Aug 27, 2021

OK, I tried to recache "category only" and everytime I try it logs me out of opencart. There may be more issues, there may even be issues with the ocmod file so go through it and debug it.

AndreyPopov · Aug 27, 2021

not sure that this construction:

Code:

            <search><![CDATA[public function recache()]]></search>
            <add position="replace" offset="139">

replace all 139 lines.

tag offset means that:
- ocmod engine find string in "search" tag
- count 139 lines from found string
- replace counted 139th line

replace block of code by ocmod possible in two way:
1. in search tag put whole block to be replaced
2. using tag regex=true with regex expression

Lee · Aug 27, 2021

I'm not sure what you are telling me but in your code you wanted to replace the entire recache() function with your version of the function and it does just that.

Edit: I think you are trying to tell me that it will only replace just one line but that's not the case, it replaces the entire recache() function with your version.

AndreyPopov · Aug 27, 2021

Lee said:

tag offset really work after replace!
but to correct replace recache() offset must be 140, not 139

Lee · Aug 27, 2021

When I read the instructions yesterday it said the number starts with 0 not 1 so 139 is correct since it's 140 lines long! In any case it works and may add or subtract an empty line but I'm not going to spend time on that!

AndreyPopov · Aug 28, 2021

found solution for "press twice before recache"

now recache start after only first click.

Advanced crawler for recache. some ideas and code

Well-Known Member

Well-Known Member

Attachments

Well-Known Member

Attachments

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Attachments

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member