by Patrik Hermansson

• Regarding passwords
• Logs where they are and how to read them
  • Read logs with the falogs command
  • The FireFlow httpd error log (/etc/httpd/logs/error_log)
  • Commands for reading logs
  • Where the logs are located
• Get back to the content menu when logged in as root
• How to show CPU, Memory and Disk use
• Handling processes
  • Verify if a specific process is running
  • Examples:
• To terminate (kill) a specific process
• To terminate (kill) all processes for analysis in Algosec
• How to terminate (kill) process for import of devices in Algosec
• How to view the version of the Algosec applications
• Restarting services
  • Restart FireFlow
  • Restart web server
  • Restart Tomcat
  • How to restart AppViz (former ABF)
• Notification configuration 2018.2 =>
• Email do not update in AFF after changes in AFA (LDAP or RADIUS) workaround

• Backup in gui won’t work (backup is already running error whith manual backup)
• To view speed and duplex on an interface
• Sync the database with the reports directory (problem with searching in objects)
• To clear type history in BASH
• To enable debug-mode in FireFlow (CLI)
• Verify that the garbage cleanup script have been running
• AFA traffic simulation in CLI
• Kill (shut down) stuck or big application queries (CLI)
• Low free disk space on / or /data partitions
• Check the status for vacuum db function (Star/stop)
• Metro service do not start "unresponcive" after breaking cluster
• A short SQL-query to get all interfaces with associated IP from all the firewalls in the map
• Licens will not install in CLI
• ABF application flows will not save
• Local account admin could not login
• Cansle connectivity check in ABF
• Art/elasticsearch/kibana backup problems
• Fetchmail troubleshooting, will not get (fetch) mails
• Menubar not showing in AppViz after upgrade to A30.10
• Verify disk speed is up to standard
• Username no longer case sensative?

• Needed ports for cluster and functions
• Where the risk profiles are located
• Regex for search in Notepad++
• How to use screens in Linux
• How to identify users in the system
• BZIP2 and GZIP archiving
• TAR commands
• How to Encrypt and Decrypt Files and Directories Using Tar and OpenSSL
• How to activate debug mode in ABF
• Boostmode on and off
• To get destination NAT from firewalls in ASMS databas
• Cluster node suddanly removed from cluster
• How the user field in ABF flows work
• How to get a session id
• How to look into .tar, .zip, .bz2 files without unpacking them
• How to clean up the session database table in postgres
• Guide for LVM on new setup virtual appliance

For the passwords, do not use »> & «< sign. Gives an error in the system. algosec_conf error=1

   [dfbf1cbc] [main           ] [2021-10-25 11:23:03,066] [INFO ] [essExecutorImpl::132 ] executing external command (UUID=aiWfbTn8): sudo -H -u afa /usr/share/fa/bin/add_del_htpasswd del 'admin'
   [dfbf1cbc] [main           ] [2021-10-25 11:23:03,545] [INFO ] [essExecutorImpl::132 ] executing external command (UUID=ukJ2DuLV): export PASSWORD=$'Kaffe&Kakor' && sudo -H -u afa /usr/share/fa/bin/add_del_htpasswd add 'admin' $PASSWORD
   [dfbf1cbc] [stderr-ukJ2DuLV] [2021-10-25 11:23:04,031] [WARN ] [pl$StreamLogger::238 ] --> sh: Kakor: command not found
   [dfbf1cbc] [stderr-ukJ2DuLV] [2021-10-25 11:23:04,034] [WARN ] [pl$StreamLogger::238 ] --> Adding password for user admin
   [dfbf1cbc] [main           ] [2021-10-25 11:23:04,064] [ERROR] [ErrorMenuItem  ::33  ] An error occurred during algosec_conf menu

   [dfbf1cbc] [main           ] [2021-10-25 11:23:04,144] [INFO ] [Main           ::97  ] Force exit from algosec_conf (exit with error 1)
   [dfbf1cbc] [Thread-36      ] [2021-10-25 11:23:04,148] [INFO ] [Main           ::90  ] algosec_conf shutdown
   [dfbf1cbc] [Thread-36      ] [2021-10-25 11:23:04,149] [INFO ] [Main           ::91  ] ------------------------------------------

Login as AFA user

su afa

If you are logged in as ROOT do the su (switch user) afa command to change user to the afa user. (this works with newer version with the root user also (as of at least A30.00 ⇒)

falogs

This is an alias that does a tail with several important logs.

Tip, use this to troubleshoot login errors in real time log.

falogs | grep {username}

The FireFlow httpd error log will give a lot of useful information when encountering problems in FireFlow

Commands:

Less 
     / = search
          /[search pattern]

Tail
     tail -n 100 [log file (full path)]
-n 100 == shows the last 100 lines from the specific file

     tail -f [filename] 
-f is "follow" i.e. appends output when the file gets bigger

–AFA logs /home/afa/.fa-history (contains lots of Info-messages, use grep -v -i ‘info’) /var/log/ /var/log/messages

–Backup logs /home/afa/backupLogs.history (up till 2018.1) /var/log/algosec-ms/ms-backuprestore.log (2018.2 ⇒ 30.10.x)

–AFF logs /usr/share/fireflow/var/log/fireflow.log /var/log/aff-boot.log

–ABF logs /var/log/bflow/bflow.log

–Other

/data/afa_catalina_base/logs/catalina.out /data/log/algosec_hadr/ (hadr, install -log and others) /var/lib/pgsql/data/pg_log/postgresql-???.log /var/log/httpd/error_log /home/afa/public_html/algosec/.ht-fa-history /etc/httpd/logs/error_log

Enter the command:

   algosec_conf

Command:

     top

Shows CPU and RAM use, press '1' to display all CPU cores, 'd' to set update interval, 'n' to set maximum tasks displayed, Shift+'P' to sort the output by CPU utilization, Shift+'M' to sort the output by Memory utilization, ‘q’ to exit

     iostat -d -x 5 3

shows the disk use

1. d == Display the device utilization report
2. x == Display extended statistics. This option works with post 2.5 kernels since it needs /proc/diskstats file or a mounted sysfs to get the statistics. This option may also work with older kernels (e.g. 2.4) only if extended statistics are available in /proc/partitions (the kernel needs to be patched for that).

5 3 == Three reports at 5 seconds intervals.

     iotop -o
shows the disk activity as top does.

1. o shows only the active i/o processes

Command:

   ps -A | grep [processnamn]
  ps -A == shows the list of running processes, -A shows all processes
  | == pipe, sends the input from the left of the pipe to the right of the pipe.
  grep == pipes the ps -A to grep, add the processname and grep sorts out and shows only those processes.

    PID		                        Process name
    27033   ?          00:00:03 	collect_gen
    27083   ?          00:00:03 	collect_gen
    27097   ?          00:00:03 	collect_gen

For a more extensive output look at the man pages for ps and grep

    ps -aef | grep collect | grep -v grep

User	PID	    PPID		                    path to the application running
Afa	    31218	31121    8 15:09 ?  00:00:02    /usr/bin/perl /usr/share/fa/bin/collect_ios -d /home/afa/algosec/monitor/CHNx_Nanj481_A_01/new_config -n CHNx_Nanj481_A_01 -m
Afa	    31530	31484   14 15:09 ?  00:00:03 	/usr/bin/perl /usr/share/fa/bin/collect_ios -d /home/afa/algosec/monitor/USAx_hous379_01_02/new_config -n USAx_hous379_01_02 -m
Afa	    31824	31791   10 15:09 ?  00:00:02    /usr/bin/perl /usr/share/fa/bin/collect_ios -d /home/afa/algosec/monitor/USAx_hous379_01_01/new_config -n USAx_hous379_01_01 -m

Command:

     kill [process PID]

This will terminate (kill) the process with the specified Process ID. Use ps -A for the process PID.

This will terminate(killall) a process but via name not PID. command:

     killall -v -u afa -e collect_ios
	-v verbose shows more output from the command
	-u specify a user, in this case user afa
	-e specify a certain search parameter, in this case "collect_ios"

pkill == as kill but use process name instead of PID.

     pkill [process name]

     pkill -9 fa_master
-9 == 

The process is import_devices

   ps -A | grep import_devices
[output with pid]
   kill -9 [pid]

When you want to kill analysis according the report no. use the below command

   pkill -9<report_no>

The below commands will display which version of each product is currently installed.

Firewall Analyzer
rpm -q fa

FireFlow
rpm -q algosec-ticketing

BusinessFlow
rpm -q BusinessFlow

AlgoSec Appliance
rpm -q algosec-appliance

Handover HA/DR from primary to secondary: Choose “13. Configure HA/DR” in the startup (content) menu – then 4. Continue according to Article Number: 303 in Algosec Knowledge Center (https://knowledgebase.algosec.com/article.php?id=303)

Restart FireFlow

restart_fireflow

Restart web server

/etc/init.d/httpd start
or
service httpd restart
or
systemctl restart httpd

Restart Tomcat

/etc/init.d/apache-tomcat start
or
service apache-tomcat restart
or
systemctl restart apache-tomcat

How to restart AppViz (former ABF) When you restart apache-tomcat AppViz will be restarted (and services connected to it) also.

     service apache-tomcat restart

Notification changed to Watchdog. Configfile : /data/algosec-ms/config/watchdog_configuration.json

   
   ! For instance backup_schedule to syslog is set to false (off) as default !

See more in the AFA admin documentation page 341

• Problem: The email address do not update from the AFA to AFF. The email address should synk automaticly but sometimes it is now working.
• Solution/Workaround: Change the user in the database with the supplyed commands. Make sure that the user is not duplicated in AFF first. (look after the user in AFF users section in configuration menu)

## Commands ### Login to the database:

psql -U postgres -d rt3 

### Look for the correct user. exchange '[userid]' with the user login name. Login name in the ASMS. For instans 'debugger' if the login username is debugger:

     select * from users where name = '[userid]';
     Verify that the email is missing or is wrong!

### Update the users email address. Replace the [email protected] with the correct email address:

     update users set emailaddress='[email protected]' where name='[userid]';

### Do “step 2” again to verify that the user is updated correctly:

     select * from users where name = '[userid]';

### Exit the postgres configuration mode

     \q

Try to restart Apache

/etc/init.d/apache-tomcat restart
or
service restart apache-tomcat
or
systemctl restart apache-tomcat

The differenses are regarding of version of ASMS.

Use the command ethtool, to see speed and duplex on the interface eth0 use the following command

ethtool eth0

Output(example):

Settings for eth0:
Supported ports: [ TP ]
Supported link modes:   10baseT/Half 10baseT/Full
          100baseT/Half 100baseT/Full
          1000baseT/Half 1000baseT/Full
          Supported pause frame use: No
          Supports auto-negotiation: Yes
          Advertised link modes:  10baseT/Half 10baseT/Full
                    100baseT/Half 100baseT/Full
                    1000baseT/Half 1000baseT/Full
          Advertised pause frame use: Symmetric
          Advertised auto-negotiation: Yes
          Link partner advertised link modes:  100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: on
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x000000ff (255)
                               drv probe link timer ifdown ifup rx_err tx_err
        Link detected: yes

Login as afa user or switch to the afa user. (su afa) run the script below. Then logout and login again.

/usr/share/fa/bin/syncDbWithReportsDir.sh

In Bash (the terminal) the commands are saved in the .bash_history file. If you write usernames and passwords in the terminal those are saved as commands in that file. In clear text! Password without echo (when you do not see the text) are not saved. Only the character that you see. To clear the file from entries, do the following. (needs to be done in all open windows (cli logins) if more than one SSH session is active/logged in)

history -c
history -w

The flag -c == clear the history list by deleting all entries.
The flag -w == writes over the .bash_history file with the empty list thereby clearing it.

The commands you write in the active (open) window will be saved in memory and written in the file on exit/logout. You can see all commands in “memory” by using the command history without any flags.

Log in to AFA as root user using SSH.
Run the following command (as a single line, linebrake after FireFlow_SiteConfig.pm, remove linebrake) to back up the existing FireFlow_SiteConfig.pm file:

cp -p /usr/share/fireflow/local/etc/site/FireFlow_SiteConfig.pm /usr/share/fireflow/local/etc/site/FireFlow_SiteConfig.pm.b4debug

Add the following lines at the end of the FireFlow_SiteConfig.pm file before the line ending with 1;

vim /usr/share/fireflow/local/etc/site/FireFlow_SiteConfig.pm

Set($LogToFile, 'debug');             
Set($LogMaxMsgLen, 0);             
Set($LogPermissions, 2);

Restart FireFlow:

restart_fireflow

Recreate the problematic scenario that you want to troubleshoot.
Download the fireflow.zip file and attach it to the support case.
In the FireFlow_SiteConfig.pm file, remove or comment with # the lines you added above to enable debugging.

Restart FireFlow:

restart_fireflow

This is to cleanup /home/afa/public_html/algosec/sessions-* folders. If that is not done the system will “run out” of inodes (check inodes load with df –ih)

df -ih

Run this to see if the garbage cleanup script have been running:

grep "clean_up_garbag.*session-" ~afa/.fa-history*[^0] | head

Example output:
[root@ASMS_2017-2 ~]# grep "clean_up_garbag.*session-" ~afa/.fa-history*[^0] | head
/home/afa/.fa-history.1:[14689] [    ]       [2018-05-05 02:00:20,203] [INFO ] [auto_remove              ::clean_up_garbag:705 ] Remove old files from directory /home/afa/public_html/algosec with prefix session-

The AFA traffic simulation feature has a CLI version.The bennefit of this compared to the GUI version is that You get much more troubleshooting info, when You have patch discovery (MAP) problems.

Log in as AFA user if not that already

su afa

Run the command test_fip with the following commands

test_fip -s [from-ip] -d [destination-ip] –o
flag -s: source
flag -d: destination
flag -o: prints fip output if given

Example:

test_fip -s 172.18.113.5 -d 172.19.150.97 –o

For instance when the application is stuck in initial plan.
In the CLI enter the commands:

ps -ef | grep 18619 
ps -ef | grep run_query

Where 18619 is the ticket ID.

Then kill those PIDs

Kill -9 [pid]

If disk utilization is high on the root ( / ) partition the system might stop working.
Also some functions like backup and report gathering might stop working if the /data partition is full. To fix (at least some of the problems) look at the KB article below.

Login is needed: https://knowledge.algosec.com/skn/tu/e15153

The following is the command to check the vacuum

grep -i vacuum /var/lib/pgsql/data/pg_logpostgresql-.log | grep  > /tmp/results.txt

i.e

grep -i vacuum /var/lib/pgsql/data/pg_logpostgresql-Tue.log | grep 2019-05-07 > /tmp/results.txt

The SSL configuration is not removed from one or all nodes.
Configure the machine_config file (/home/afa/.fa/machine_config)

vim /home/afa/.fa/machine_config

Change secure_conection=true => secure_conection=false

Restart apache-tomcat service

service apache-tomcat restart  (systemctl restart apache-tomcat)

In CLI on the AlgoSec server run:

sqlite3 /home/afa/.fa/map.sqlite "SELECT DeviceName, HwName, IP FROM Interface INNER JOIN Device ON Device.DeviceID = Interface.DeviceID WHERE DeviceName IS NOT NULL AND DeviceName != \"\" AND IP != \"None\";" ".quit"

Edit: this command will also show subnet-ID with CIDR-mask.

sqlite3 /home/afa/.fa/map.sqlite "SELECT DeviceName, HwName, IP, CIDR FROM Interface INNER JOIN Device ON Device.DeviceID = Interface.DeviceID INNER JOIN Subnet ON Subnet.SubnetID = Interface.SubnetID WHERE DeviceName IS NOT NULL AND DeviceName != \"\" AND IP != \"None\";" ".quit"

For version 2018.1.x-x
If this occures on the second device in a DA/HA cluster check if the metro service is running.
Usually not, and to start you need to start the apache-tomcat service.

service start apache-tomcat or systemctl start apache-tomcat.service 

Rememeber to shut it down after the installation of the license.

service stop apache-tomcat or systemctl stop apache-tomcat.service

This could be because of not allowed text (text that can be translated into code) in some of the comments fields. Affected fields:

Custom fields 
Flow Names 
Comments

Algosec KB for this: https://knowledge.algosec.com/skn/tu/e16448

Example on what is considered as code:

<script>
</script>
src="*"
eval(*)
expression(*)
javascript:
vbscript:
onload*=

also avoid write <*> html tags 
 
"*" means anything between

Check the followin KB: https://knowledge.algosec.com/skn/c6/AlgoPedia/e4998/Login_Failed_incorrect_user_name_or_password

If that looks ok, check if two or more accounts have the same password. Is possible if new user are added via the users_info.xml file.
If that is the case remove the other account or change the email to specific emails for all accounts.

There MAY be a workaround. It isn't verified, but we have used it in other occasions when an ABF application seems to be stuck in updating:

First, Connect to postgres

root@ITSEELM-BB4261:~#psql -U postgres
Password for user postgres:
psql (9.2.5)
Type "help" for help.

Enable prettyprint & connect to the bflow database

postgres=# \x on

Expanded display is on.

postgres=# \c bflow

You are now connected to database “bflow” as user “postgres”.

Select the application with the ID. This can be found in the URL. Example:

"https://fo.ikea.com/BusinessFlow/#/application/ --> 2797 <-- /dashboard"

bflow=# select * from applications where id=2797;

-[ RECORD 1 ]------------------+---------------------------
id | 2797
app_id | 2433
creation_ts | 2019-04-01 11:45:02.110999
lcname | mfc-le-sto-371
name | MFC-LE-STO-371
update_ts | 2019-04-01 11:48:33.302
connectivity_id | 85479
metadata_id | 1100
revision_id | 2827
connectivity_scan_in_progress | f
vulnerability_scan_in_progress | f
last_risk_check |
risk_scan_in_progress | f
risk_score |
risks_information_up_to_date | f
discovery_update_in_progress | f

This gives us some data to look at. What we need for next step is the APP_ID field. Use this field in the next query:

bflow=# select * from application_metadata where appId=2433;
-[ RECORD 1 ]------+---------------------------
id | 1100
appid | 2433
applicationlock | t
creation_ts | 2019-02-20 12:53:10.764664
update_ts | 2019-02-20 12:53:10.817
lifecyclephase_id | 1
name_sequence | 1
expiration_date |
rename_in_progress | f

This shows us that the applicationlock field is indeed TRUE. set this to FALSE using the APPLICATION_METADATA ID, not the application ID:

bflow=# update application_metadata set applicationlock=false where id=1100;
UPDATE 1

Verify that the flag is correct (False):

bflow=# select * from application_metadata where appId=2433;
-[ RECORD 1 ]------+---------------------------
id | 1100
appid | 2433
applicationlock | f
creation_ts | 2019-02-20 12:53:10.764664
update_ts | 2019-02-20 12:53:10.817
lifecyclephase_id | 1
name_sequence | 1
expiration_date |
rename_in_progress | f

Quit when done.

bflow=# \q

The new backup needs the elasticsearch service to be running. It will fail othervice. (Version 2018.2 ⇒ )
In at least 2018.2.870 - 2018.2.900 there wherer a mismach with the versions of elasticsearch and kiban4. This could generate problems with the services and backups.
The problem is resolved in version 2018.2.900-xyz (accoring to Algosec)

Elasticsearch:
Check the service via:

service elasticsearch status

or

systemctl status elasticsearch

Check that the service starts with the system

chkconfig | grep -i elasticsearch

Kibana:
Check the service via:

service kibana4 status

or

systemctl status kibana4

Check that the service starts with the system

chkconfig | grep -i kibana4

There is a script to start the services (or stop them) and enable the start with the system.
Script: toggle_art.sh
To run it:

/usr/share/fa/bin/toggle_art.sh

use the on/off to turn it on or of… /usr/share/fa/bin/toggle_art.sh on or

/usr/share/fa/bin/toggle_art.sh off

Log: /var/log/fetchmail.log
Symptom: The system does not get (fetches) emails from email server.
Test the function with the following command:

/usr/bin/fetchmail -c -v -p POP3 -P 995 --ssl -u [username] -L /var/log/fetchmail.log [server FQDN/ip]

Exchange POP3 and 995 –ssl if needed. Enter username and server FQDN or ip.
If that works follow the checklist below. If not check the logfile to see what went wrong. Check:
That the ownership are correct on the file, .fetchmailrc. (fireflow should be owner of the file) as user root do:

chown fireflow:fireflow .fetchmailrc
        

That the rights are correct on the file, .fetchmailrc (chmod 0700)
as user root do:

chmod 0700 .fetchmailrc

Login at fireflowuser (su - fireflow), test with

/usr/bin/fetchmail

without any more inforamtion. The command get the rest from the .fetchmailrc file in /home/fireflow directory. Remember that . files are hidden files.

If it gives an error make sure that point 1 and 2 are done and that the .fetchmailrc file is correctly filled in. In one instance the file needed to be redone to get it working.

Problem:
After upgradeing to version A30.10 the AppViz (fd Businessflow) menu (blue top row) does not show.

Troubleshooting:
When checking in the webbrowser the URL gave wrong rederict URL. For us the domain was missing.

Solution:
Check in the AppViz config file. (/home/bflow/config/user.properties)
The following parameters needs to be populated with the full url:

afa.hostname=****************** (removed for the document)
fireflow.hostname=****************** (removed for the document)

The disk speed (read/write) in MB/s
Below is built in check for upgradeing the system.

Is blocked User approval Is allowed

               needed \\

0——80—— 80———100—————————–300————— =⇒

So to have a good system at least there should be R/W of 300MB/s

How to check this?
Linux tools hdparm (for read) and dd (for write).

Disclamer!
Not sure how much this will affect the system so do this outsid of working hours for production.
!Disclamer

hdparm -Ttv [partition, like /dev/sdb1]

dd if=/dev/zero of=/var/wrtest/test oflag=direct bs=128k count=32k

if = input file, of = output file. Put it someherer the system can write ~4GB (in this setting). Remember to remove the of file.

On my home system with older SATA disk i got:

root@system#dd if=/dev/zero of=/var/wrtest/test oflag=direct bs=128k count=32k

32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 53.6856 s, 80.0 MB/s

So 80.0 MB/s in write speed.

System Version: A32.0.x-y

One incident when user could not login, had a duplicate user when one was all lower case and one user with first letter upper case. The system tried to match with the first username in the list and that was the one with upper case. Not the actual account and the user could not login.

Solution?:
Ended up deleting the account with upper case first letter and user could login again after that.

If possible for uptime allow any ports between the nodes in HA.
Twice i have had problem at customer due to the fact Algosec did not update there documentation for needed ports.
Not in documentation nor in KB. (Custemer have PVLAN with ACL)

(not in use since 2018.1)


Ports required for communications (central manager, remote agents) in geo-distributed architecture.
1. For configuration procedures (adding a remote agent, adding/editing/deleting devices) that must be synchronous:

2. For log collection, monitor and data collections procedures that may be asynchronous:

3. For communications between master-slave in load-distributed architecture:

/home/afa/.fa/risk_profiles/

Files are saved as *.xml

To search after ip-addresses in Notepad++ use the following regex:

[0-9]+.[0-9]+.[0-9]+.[0-9]+
or
\b(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(?1)){3}\b