Stream SonarQube Logs to Aria Operations for Logs using Fluentd

PAGE CONTENTS
Goal
Applicable Environment Details & Assumptions
Step-by-Step Guide

Goal

Stream SonarQube Logs to Aria Operations for Logs using Fluentd Agent

Applicable Environment Details & Assumptions

SonarQube Community Edition 8.9 LTS
Ubuntu 16.04.1 LTS (Xenial Xerus)
Access to VMware Cloud Services, Aria Operations for Logs (vRealize Log Insight Cloud)

Step-by-Step Guide

Installation of Fluentd requires Ruby version >= 2.6.0
I will choose rbenv package manager to install 2.6.0 ruby on Ubuntu. These managers help us switch between required versions of Ruby.
Install dependencies for ruby and rails

sudo apt install curl
curl -sL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

sudo apt-get update
sudo apt-get install git-core zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev software-properties-common libffi-dev nodejs yarn

4. Now let us install version manager rbenv. First, let us install ‘rbenv’ and then ‘ruby-build’

cd
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
exec $SHELL

git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
exec $SHELL

5. Now let us install the required ruby version i.e., 2.6.10, and activate v2.6.10 as the new default run the below commands.

rbenv install 2.6.10
rbenv global 2.6.10

6. Install the bundler

gem install bundler
rbenv rehash

7. Install Fluentd using gem software package

gem install fluentd

8. Download and install the Fluentd Aria Operations for Logs (vRealize Log Insight Cloud) output plugin. [You can find this in VMware Cloud Services > vRealize Log Insight Cloud > Log Sources > Fluentd]

wget https://github.com/vmware/fluent-plugin-vmware-log-intelligence/releases/download/v2.0.6/fluent-plugin-vmware-log-intelligence-2.0.6.gem && \
gem install fluent-plugin-vmware-log-intelligence-2.0.6.gem

9. Navigate to the Fluentd configuration file located in the system (/etc/fluent/fluent.conf). Create the config file manually if it does not exist. Add the following input configuration.

<source>
  @type tail
  path <input_file_path>
  pos_file <input_file_log_path>/input_file_log_position.pos
  read_from_head true
  tag <tag_name>
  format none
  time_format %d/%b/%Y:%H:%M:%S %z
</source>
  
<match <tag_name>>
  @type vmware_log_intelligence
  endpoint_url <ENDPOINT_URL>
  verify_ssl true
  <headers>
    Content-Type application/json
    Authorization Bearer <API Key>
    structure simple
  </headers>
  <buffer>
    @type file
    path /path/to/buffer/lint.buf
    chunk_limit_records 300
    flush_interval 20s
    delayed_commit_timeout 120s
    retry_max_times 3
  </buffer>
  <format>
    @type json
    tag_key text
  </format>
</match>

<input_file_path> : Complete path to input log file.
<input_file_position_path> : Any directory for which the relevant user has read and write permissions (Recommendation: Use the same path that stores input log files).
<tag_name> : Specify a tag name to be applied for all the logs of input file.

<API Key> :Generate a vRealize Log Insight Cloud API Key from here or use an existing Key.

<ENDPOINT_URL> : https://data.staging.symphony-dev.com/le-mans/v1/streams/ingestion-pipeline-stream.
For more information about these parameters and the plugin, see the github repository.

10. My fluent.conf looks like this : For your ready reference