This article contains commands to identify causes of incorrect operation of the platform, as well as commands to restart the platform and some services to restore their work. To run commands, connect to the server with the platform via SSH.

Some commands may require superuser privileges.

General diagnostics

The section contains a list of commands for the first diagnostic step. These commands will eliminate basic problems and reduce troubleshooting time.

Operating system (OS) version

If the OS of the platform server is not supported, the installation will end with an error or the platform will not work properly. To determine the OS version, run the command:

cat /etc/*release
CODE

See BILLmanager 6 documentation for a list of supported operating systems:

Server date and time

During periodic synchronization with the license server, the date and time are checked. If an incorrect date or time is set on the server with the platform, the platform will be blocked or its operation will be incorrect. To determine the current date and time on the server, run the command:

date -R
CODE

Disk space and RAM usage

For the platform to work correctly, the free disk space and RAM must meet the requirements specified in the BILLmanager documentation:

Also, backups will not be created if there is not enough free space or RAM. To check the amount of used disk space and file system data, run the command:

df -hT
CODE

To check the information about RAM, run the command:

free -h
CODE

Inodes

Inodes is the structure in which file metadata is stored. The platform will not work correctly if the server is out of inodes, even if there is free disk space. Typical behavior when there is a shortage of inodes includes reduced performance, inability to create files, incorrect output of information in the platform interface. To check the number and proportion of inodes used in the filesystem, run the command:

df -i
CODE

Troubleshooting Linux systems

Examine the system logs to troubleshoot your Linux system. Below are tools for troubleshooting and searching for errors in the server logs.

Circular kernel buffer

One way to find out if the system is working incorrectly is to look at the kernel log using the dmesg utility. The kernel records all events in a circular buffer while the system is booting and running. dmesg  will allow you to examine the kernel messages and also identify hardware-related problems. To search for problems, run this command:

dmesg | grep -i -E 'error|failed|critical|bug|panic'
CODE

The journalctl utility

You can use the journalctl utility to analyze the logs and detect system problems. The utility displays the logs of the Linux system services. To detect abnormal behavior of a Linux system, run the command:

journalctl | grep -i -E 'error|failed|critical|bug|panic'
CODE

Licensing

Trial license

Trial license is intended for getting acquainted with the product and can be activated only once on one server. Also, you cannot activate the trial after the commercial version. To check if a BILLmanager license was activated on the server earlier, run the command:

cat /usr/local/mgr5/var/licctl.log | grep 'Got'
CODE

Linking a license to a server

If the synchronization with the license server is lost, the platform will not start. To check the license information on the server, run the command:

/usr/local/mgr5/sbin/licctl info billmgr
CODE

If the output does not show the license number, it means that the license is not linked to a server. To activate the license on the server, run the command:

/usr/local/mgr5/sbin/licctl fetch billmgr
CODE

If you get the error Failed to get license when trying to activate the license:

  1. Generate a new license key in your client area. Enter Products/Services → ISPsystem licenses → select a license → click Edit → click Generate new key.

  2. Activate the license with the command:

    /usr/local/mgr5/sbin/licctl fetch billmgr KEY
    CODE
    KEY —  new key value

The default key generation limit is 1 time per month. If you have reached the limit, contact ISPsystem Technical Support to generate a new key.

For more information about activating a license using the key, see the Activating a license with a key in the BILLmanager documentation.

If generating a new key does not help, check the /usr/local/mgr5/var/licctl.log for diagnostics.

Payments

If a payment is successfully credited on the payment system side and its status has not been updated in BILLmanager, start manual synchronization with the payment system to update the payment status. To do this, run the command:

/usr/local/mgr5/sbin/mgrctl -m billmgr paymethod.checkpay
CODE

Restarting the platform

Restarting the platform can solve a number of basic problems, such as freezing, incorrect display of information in the interface, incorrect behavior of service processing modules. Also, restarting the platform is necessary to save the settings of payment modules, and service processing modules, when you change the logging level and other settings. To restart the platform, run the command:

/usr/local/mgr5/sbin/mgrctl -m billmgr -R
CODE

The operation of the platform will be stopped while the restarting is in progress. 

Logging

This section gives an example of diagnostics with the main log of the billmgr.log platform. Other logs may also be required for diagnostics. For example, in case of payment problems, information about errors will be recorded in the logs of the payment module, and in case of problems with reselling services — in the log of the interaction between the selling and reselling billing systems. The list of all logs is located at /usr/local/mgr5/var/. For the names of the logs, see the corresponding section in the BILLmanager 6 documentation. For example, logs to diagnose Paymaster payment problems are listed in the PayMaster article.

The main log of the platform billmgr.log records all the main events and errors. To check for errors in the main log, run the command:

grep 'ERROR' /usr/local/mgr5/var/billmgr.log
CODE

To check the logs at the moment, run the command:

tail -f /usr/local/mgr5/var/billmgr.log
CODE

Any problem can be diagnosed using the command to display the logs at the moment. To do this:

  1. Open the section in the BILLmanager interface where the incorrect behavior occurs.
  2. Connect to the server with the platform via SSH.
  3. Open the real-time log with the command tail -f /usr/local/mgr5/var/billmgr.log.
  4. Reproduce the incorrect behavior in the interface.

As a result of these actions, the log will display an error and the request that precedes it. This will help identify the cause of the incorrect behavior.

For detailed diagnostics, the above instructions can be used with any other log, such as the processing module log. 

Threads and increased logging level

You can find out more information about the error by increasing the logging level and checking the log thread with the error.

The instructions below include restarting the platform.

To get additional information:

  1. Increase the logging level for the log in question. To do this, add the line "name_log.* 9" to the file /usr/local/mgr5/etc/debug.conf.

    name_log — the name of the log before the dot. For example, for billmgr.log you should enter billmgr

  2. Restart the platform:

    /usr/local/mgr5/sbin/mgrctl -m billmgr -R
    CODE

  3. Identify the thread with the error. To do this, open the required log with a search by the error:

    grep 'ERROR' /usr/local/mgr5/var/name_log.log
    CODE

    name_log.log — the name of the log

    The output will show the error thread. Example of response:

    Mar  6 18:33:01 [483562:72787] libmgr ERROR Error: Type: 'missed' Object: 'function' Value: 'payment_cash_register.check'
Mar  6 18:36:01 [483562:72797] libmgr ERROR Error: Type: 'missed' Object: 'function' Value: 'payment_cash_register.send'
    CODE

    483562:72787 —  log thread

  4. Check the log file on the thread found:

    grep '483562:72787' /usr/local/mgr5/var/name.log.log
    CODE

    This command will display detailed information about the error.

  5. After diagnostics, reset the logging level to the default so that the detailed logs do not take up free space. To do this, delete or comment the line "name_log.* 9” added to the file /usr/local/mgr5/etc/debug.conf.

    name_log — the name of the log before the dot

Searching for information in the database

The section contains basic database (DB) queries. To make other queries, use data about the BILLmanager database structure. For more information about the database structure, see the article Database structure in the BILLmanager 6 documentation.

There are potential risks involved in tampering with the DB. We do not recommend making manual edits to the database, as it can disrupt the correct operation of the platform.

Any actions with the database should be performed only after backing up the platform. 

To query the database, connect to it with a command:

mysql billmgr
CODE

Referral program

If a promo code is not activated, a query to the DB will help reveal information about the client's attachment to the referral program. To display information on the client's referral program, run a query:

Select * from account2project where account='ID_client'\G;
CODE

ID_client — client ID in BILLmanager

The query will display information about:

  • the partner program that the client uses from the provider;
  • the partner from whose link the client was attracted;
  • the referral program, through which the client is attracted.

Service processing modules

If an incorrect processing module is assigned to a service or no processing module at all is asigned, there will be problems with activating or deleting the service. Also, a "reset” of the processing module may occur when a service is transferred within a reselling billing service.

To check the service processing module, run the query:

select processingmodule from item where id =<item_id>;
CODE

item_id — service ID in BILLmanager

License details

The query to collect license data will display all license parameters, including the name, remote ID and activation key. To display the license parameters, run the query:

select * from itemparam where item=<item_id>\G
CODE

item_id — service ID in BILLmanager

The resulting output will display the data:

  • intname - the name of the parameter. For example, lickey is the name of the parameter for the license key;
  • value - the value of the parameter. For example, lickey will display the value of the key, and import_remote_id will display the value of the deleted identifier.

Provider

Query to the database will allow you to get the current status of the provider, linkage of a tariff plan to it and other parameters.

The queries are valid for the BILLmanager Hosting&Cloud edition.

To check the active providers, run the query:

select id, active from project\G;
CODE

To find out which provider the tariff plan is linked to, run the query:

select project from pricelist where id in (select pricelist from item where id='<item_id>');
CODE

item_id — ID услуги в BILLmanager

Filters

With the help of queries to the database, you can apply filters that are not in the interface of the platform. This section contains queries to display data on customers with an active Promised Payment and a non-zero credit limit.

To display a list of services for which a promised payment was activated, run the query:

SELECT * FROM promisepayment;
CODE

To display the list of clients with an active credit limit, run the query:

select account from subaccount where creditlimit>0.0000
CODE