Link Checker for XenForo 2.x by AddonsLab 3.5.6

[valaiVerified member]

Compatibility

XF 2.1 / 2.2

Description:
The add-on gives full control over the links you have in all your posts, allowing you to quickly search for them, check their status, edit them one by one or at a time. Support for other content types (resources, profile posts, third-party blog, article add-ons, etc.) will be released as separate paid plugins. Please see the related add-ons section for more information.

Supported BB Code tags are URL, MEDIA and IMG. Supported media types are YouTube, DailyMotion and Vimeo. Other media types can be supported if needed.

The product comes with the following main features and pages :

1. A new section in Admin Panel, under Tools section, allowing you control the product.
2. Rebuild page to find all links posted on your board currently. A separate rebuild task can be run to check the status of the links.
3. Manage Links page lists all links found so far. It has a quick search and filter controls allowing you to quickly find links by URL.
4. Each link can be edited individually. Edit page gives full information about last status check, allows to force-update the status by checking it one more time, to set the URL and text of the link or to delete it. It also has a link to the content where the URL was used so you can check the post quickly if needed.
5. Deletion of links allows setting a placeholder message that will replace the link. It can be left empty for the simple removal of the link.
6. Advanced batch update tool allows you to change an unlimited number of URLs at once. Caution: use the tool with care as it modifies the database content, and any submitted changes will not be reversible.
7. Batch update page allows to quickly find any link by its status code, the content of the URL or its text, URL domain etc. Replacement options are to replace the URL/text entirely, or to do partial replacement only replacing the parts defined in your search criteria. The tag can also be deleted or the content having the tag in it (post, thread) can be deleted entirely.
8. Regular expressions are supported in both search and replace mode. Subpatterns of matched searched criteria can be used as variables in the replacement.
9. Live preview of all changes, which will be saved to the database.
10. Ability to process only sevaral of tags using the checkboxes in the preview.
11. A link can be shown under each post allowing to force-update the status of all links found in the post. Controllable by a new usergroup permission.
12. Moderation queue support. Invalid links are added to the queue where forum moderators can see the links detected in their forums and modify/delete them. Their actions are logged in Moderator Log. The permission should be enabled on per-usergroup basis.
13. Alert users about their link not being valid. The permission should be set on per-usergroup basis. The alert is automatically removed if the URL is validated later.
14. Links in the posts can have a "valid" or an "invalid" icon next to the link based on their status. The usergroups allowed to see the icons are controlled via XenForo Usergroup Permissions page.
15. The "valid" icon can be hidden for all usergroups at once if you prefer not to have it but to show broken links only.
16. Special support for "429 Too Many Requests" URLs. If a server blocks the requests because they are too often, the validation is scheduled and run with some delay.
17. Batch Update and Dead Link Check can run both in the browser and as a cron job in background. In that case, admins can see the status of the background tasks on that pages.

Here is a video demonstrating the Batch Update tool in Admin Panel (made on XenForo 1.x version but the interface is identical on XenForo 2.x)

We are getting reports from our customers, that several URLs are actually alive but the product does not recognize it and gives an error when trying to save the URL in Admin Panel.
It is important to realize, that checking a URL automatically is not the same as visiting the URL in your browsers. There are many techniques available, that can make a page to work when visiting by the browser, or when Google indexes the site, but block any other bots, including our bot that checks the status of URL.
Feel free to contact us whenever you get any URL that doesn't work for you in our add-on but works via browser, but so far in all cases, we had this resulted from the site/IP being blocked by another site or automated requests being blocked altogether. Please consider adding the URL/Domain to the exclusion list in these cases.

Disclaimer: Please make sure to backup your database before using the tool. It is processing the real content of the database and any changes committed are impossible to revert.

Planned Features :

1. Front-end management of broken links by users who posted them.

Plus d'info :
Link Checker for XenForo 2.x by AddonsLab

Add-On Information The add-on gives full control over the links you have in all your posts, allowing you to search for them quickly, check their status, modify them one by one or at once. Support for other content types (resources, profile...

Spoiler: FAQ

Question: Are there any steps to follow after the first installation?
Answer:

1. Setup the permissions of who can see link status in posts - Usergroups -> [Choose the usergroup] -> Link Checker
2. Trigger full indexation of all existing links - Tools -> Links -> Rebuild Data ->

Index Links. This process finds all links in the database and may take some time to execute. It does not check the status of links.

1. Check the status of all links using the "Check Dead Links" tool on the same page. This tool sends requests to each URL to identify if they are broken or no. It will take a long time to execute, especially with many links and if many of them are broken or are pointing to slow servers.
2. Once the task above is completed you will be able to see the list of links, filter them by status and apply batch modification actions from Batch Update page. For the list of all links with the ability to edit each of them use Manage Tags page.
3. Check product settings at Setup -> Options -> Link Checker by AddonsLab

Disclaimer: Please make sure to backup your database before using the tool. It is processing the real content of the database and any changes committed are impossible to revert.

Question: Does the add-on handle properly the URLs containing Unicode characters?
Answer:
For all admins working on boards using Unicode and other non-URL compatible characters in URLs we would like to stress the importance of understanding, how URLs containing special characters work. URLs themselves are not allowed to have Unicode/special characters. All Unicode characters are converted to their compatible version (URL encoding). You can confirm this by editing any post, which has a Unicode-containing link and see the source code. And you will see the link does not contain Unicode characters and is fully encoded. Basically, our product just takes the URLs from the post and processes them just as they are in the text. If a user follows the URL, browsers show the text in Unicode characters, but this does not mean, that the URL actually has this characters inside. We have decided, however, to avoid confusion for admins, to show decoded characters on the list and Batch Update page. When you are editing a link, however, we will show the real URL, and any URL you paste as a replacement should be properly encoded. You should not try to type Unicode characters in the field directly, but rather copy the link from a browser, which will automatically do the conversion for you.

Here is the list of processes, where we do URL encoding conversion:

1. In Manage Links page, you will see the Unicode characters inside URLs.
2. When editing a link, you will see the Unicode characters in Full Code sector but will see the encoded version in the field to edit the URL.
3. On Batch update page, Search Link URL content will be automatically URL-encoded before the search. So the text you type there can be in Unicode or contain special characters, they will be converted to URL format on the fly.
4. Process URL field content will be transformed into URL-encoded version only if you use "Replace found part" mode. So, if using "Replace found part" mode, you should use Unicode characters in both search and replacement field. However, if you are using "Replace Fully" mode, the URL you will use in the field should be fully valid, meaning all Unicode characters should be already converted to URL-compatible version.
5. Regular expression matching is done after the search term is URL-encoded.
6. Subpattern replacement is done before the replacement term is converted to URL-encoded version.
7. In replacement preview, you will see the Unicode characters, but in actual replacement will use valid URL-encoded strings.

All admins dealing with Unicode URLs are strongly advised to backup the database before using the tool. We recommend testing batch replacement with one-two URLs only, checking the actual post content, and making sure the replacement was processed as expected.

Question: What is the User Agent option and how should I use it?
Answer:
User Agent is a string identifier, that identifies your board for the remote sites, for them to know "who" makes the request. By default, the "User Agent" contains your board URL. In some cases, this identifier is used on remote sites to block your website from making further requests to them. You can change the user agent used in this option, but you should do so at your own risk, as changing of the user agent to overcome site access limitations may be illegal in some cases. User agent format is usually ProductName/number.number (any comment here).

Question: Why I see Timeout status on some links sometimes, and the same links work in another case?
Answer: Timeout status means, that the request to check link's validity timed out, meaning the remote server did not reply in time. This can happen, when, for example, the remote server is down for some time. It also can be due to the fact, that links are checked "quickly" when the user posts them, with 0.5-second timeout, but the links are scheduled for more thorough check, with 5 seconds timeout, via a deferred task. So, you may see a link as invalid right after posting, but after some time, if the link is valid, it will turn to valid status.
Some remote servers prefer not to send 403 Forbidden error when the access to some resource is denied (e.g. you IP is blocked on their server), but to block the request by keeping it open long time and eventually timing out. In this case you will get permanent timeout status even if you re-check the link.
Rebuild Data admin page also allows to manually set the number of seconds for a request to timeout. If you see too many timeout errors, you should consider to increase this limit. We have also implemented the option to check only timed-out links, so we would recommend to check all your links with normal timeout limit (1 second), and only after that check the links that timed-out with higher timeout limit (e.g. 5 seconds).
Please note, that timeout error is a connectivity issue and not a bug in the product, and if the same link times out one time and not another time with all other conditions being equal, that does not mean the product works differently each time, but that the link is available for not at that moment.

Question: Why I see some URLs as valid, but visiting them I see they dead/not found?
Answer: The product depends on information provided by the remote server in page headers when checking the status of a link. In the case described, that means the server returns an error page to users, but in headers it actually returns a success code.

Version 2.9.0 Unfurled URL support fixed on XenForo 2.1.x

This version fixes URL unfurl support for XenForo 2.1.x. We found out these URLs were not detected and processed by the add-on. This is fixed now, and the URLs will be listed in the Admin Panel and checked after this version is installed and the URL index is rebuilt.

This release also fixes the issue with the icon of a link showing its status is under the block of unfurled URL, instead of being shown next to the link in the block.

In this version, we also implemented special support for missing files at Turbobit.net | Unlimited and fast file cloud which returns "200 OK" status instead of "404 Not Found" status for wrong URLs. For this specific case, we will look for special page elements on the page, and if the file was not found we will use the "404 Not Found" status as expected.

The new version is available for all licensed customers at

Version 2.9.1 Not found page support for uptobox.com :
The release adds special support for the domain uptobox.com which in some cases does not provide correct response code for deleted/not found files. It also fixes a PHP error from raised on Action log and Manage links pages introduced in the latest version and corrects the breadcrumbs to these pages.

Version 3.0.0 XenForo 2.2 compatibility update : This version implements compatibility with XenForo 2.2.

Version 3.1.2 Bug-fix : Invalid license error for XenForo 1.x licenses
The release solves the Invalid license error received by owners of a license issues for XenForo 1.x version of this product.
The update requires [AL] Core Package version 1.2.3+
The new version is available for all licensed customers at

Version 3.2.0 Delete/Replace tag as tag replacement option
The add-on is enhanced to allow replacing a broken tag with another tag. For that, use the option "Delete/Replace Tag". You can provide any text/BB Code as a replacement, and if you prefer keeping the URL in some form, just use {link} as a placeholder, e.g. [DMLURL]{link}[/DMLURL] will replace the broken link with BB Code DMLURL wrapping the link.
This release also fixes an error during tag indexation, for tags with attributes longer than 255 characters.
We have also added a missing phrase allm_tag in the moderation queue filter.
In case you experience performance problems with approval queue and dead links, please disable dead link moderation from the product's options page. If it is already disabled, enable it, save the options, and disable it again. This will remove dead link records from the moderation queue and rebuild the moderation queue cache.

Version 3.2.2 Stability fix : prevent indexation failure of a tag is invalid and log an error instead
With this release, we made sure the indexation of links does not fail if one of tags is invalid and can't be indexed. An error will be logged in the admin panel with information required to check the content of the link manually.

Version 3.2.3 Direct link to visit the link from the preview
In this version some minor enhancements have been implemented:
1. Replacing a link with its text deletes the link completely if the text itself is a link
2. A link is added in the Preview to visit the link directly

Version 3.2.6 :
This version fixes a PHP error ErrorException: Template error: [E_WARNING] Attempt to read property "serialized_links" on null src/addons/AddonsLab/LinkChecker/XF/Entity/Post.php:43.

Version 3.3.1 :
This version fixes the problem with the action log list filter. Now it is possible to search the logs by old and new URL/Text.
The new version is available for all licensed customers at Login - AddonsLab

Version 3.4.0 New feature: Restoration of deleted links :
This release implements a feature to restore deleted links with their original version. Logs that are applicable show a Restore link on the Action logs page. The page also allows batch restoration of multiple deleted links by log ID or by domain match.
If the links have been removed without any unique replacement value, it is impossible to detect the original location in the post where the links were before deletion. The product provides an option to set the name of a backup table, that contains the original versions of the posts. We try to use the original text to detect the location of the deleted link.
If the location of the link is impossible to detect, it is added to the post, at the end of the text.
Restore action shows a preview window, where you can see the post content before and after restoration.

Restoration of deleted links is a destructive operation and may irreversibly modify post content, especially if you apply batch restoration. Make sure to backup your board before any batch actions. No support will be provided in case of data loss or damage as a result of batch replacements and restoration.

Version 3.5.5 New search criteria - Filter links that redirect :
- The new versions adds the ability to filter the URLs depending on their redirect status to apply special batch update operations to these URLs only.
- The new version also fixes an error in status checker. Previously, you might have noticed there are some URLs with 30x status. Normally, these redirect should be followed and show the final status, not the redirect status, but this could fail in some cases.

Version 3.5.6 Batch update confirmation :

- In this release, we have implemented additional confirmation during Batch Update of links.
- The new version is available for all licensed customers at Login - AddonsLab

 

[XTR] External Links Redirect Warning  1.0.2

AndyB Forum cookie  1.3