This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Module: FERRET

Pigsty has built-in FerretDB support, which is a MongoDB compatiable middleware based on PostgreSQL.

MongoDB was once a stunning technology, allowing developers to cast aside the “schema constraints” of relational databases and quickly build applications. However, over time, MongoDB abandoned its open-source nature, changing its license to SSPL, which made it unusable for many open-source projects and early commercial projects. Most MongoDB users actually do not need the advanced features provided by MongoDB, but they do need an easy-to-use open-source document database solution. To fill this gap, FerretDB was born.

PostgreSQL’s JSON functionality is already well-rounded: binary storage JSONB, GIN arbitrary field indexing, various JSON processing functions, JSON PATH, and JSON Schema, it has long been a fully-featured, high-performance document database. However, providing alternative functionality and direct emulation are not the same. FerretDB can provide a smooth transition to PostgreSQL for applications driven by MongoDB drivers.

Pigsty provided a Docker-Compose support for FerretDB in 1.x, and native deployment support since v2.3. As an optional feature, it greatly benefits the enrichment of the PostgreSQL ecosystem. The Pigsty community has already become a partner with the FerretDB community, and we shall find more opportunities to work together in the future.

Currently, FerretDB 2.0 provides a new version based on the Microsoft DocumentDB extension plugin, which is now officially supported in Pigsty v3.3.

1 - Usage

Quick start guide for FerretDB - how to get started, connect reliably, and use the mongosh client tool

Install Client Tools

You can use MongoDB’s command-line tool MongoSH to access FerretDB.

Use the pig command to add MongoDB repository, then install mongosh using yum or apt:

pig repo add mongo -u
yum install mongodb-mongosh
apt install mongodb-mongosh

Connect to FerretDB

You can access FerretDB using MongoDB connection strings with any MongoDB driver in any language. Here’s an example using the mongosh CLI tool:

$ mongosh
Current Mongosh Log ID:	67ba8c1fe551f042bf51e943
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.4.0
Using MongoDB:		7.0.77
Using Mongosh:		2.4.0

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

test>

Authentication

You can log in with different users. See FerretDB: Authentication for details.

mongosh 'mongodb://dbuser_meta:[email protected]:27017/meta'      # Business admin user
mongosh 'mongodb://dbuser_view:[email protected]:27017/meta'    # Read-only user

Quick Start

You can connect to FerretDB and use it as if it were a MongoDB cluster.

$ mongosh 'mongodb://dbuser_meta:[email protected]:27017/meta'

MongoDB commands are translated to SQL and executed in the underlying PostgreSQL:

use test                            // CREATE SCHEMA test;
db.dropDatabase();                  // DROP SCHEMA test;
db.createCollection('posts');       // CREATE TABLE posts(_data JSONB,...)
db.posts.insertOne({                // INSERT INTO posts VALUES(...);
    title: 'Post One',body: 'Body of post one',category: 'News',tags: ['news', 'events'],
    user: {name: 'John Doe',status: 'author'},date: Date()}
);
db.posts.find().limit(2).pretty();  // SELECT * FROM posts LIMIT 2;
db.posts.createIndex({ title: 1 })  // CREATE INDEX ON posts(_data->>'title');

If you’re not familiar with MongoDB, here’s a quick tutorial that works with FerretDB: Perform CRUD Operations with MongoDB Shell

To generate sample workload, you can use this simple test script with mongosh:

cat > benchmark.js <<'EOF'
const coll = "testColl";
const numDocs = 1000;

for (let i = 0; i < numDocs; i++) {  // insert
  db.getCollection(coll).insertOne({ num: i, name: "MongoDB Benchmark Test" });
}

for (let i = 0; i < numDocs; i++) {  // select
  db.getCollection(coll).find({ num: i });
}

for (let i = 0; i < numDocs; i++) {  // update
  db.getCollection(coll).updateOne({ num: i }, { $set: { name: "Updated" } });
}

for (let i = 0; i < numDocs; i++) {  // delete
  db.getCollection(coll).deleteOne({ num: i });
}
EOF

mongosh 'mongodb://dbuser_meta:[email protected]:27017' benchmark.js

You can check FerretDB’s supported MongoDB commands and known differences. For basic usage, these differences are usually not significant.



2 - Configuration

Configure FerretDB cluster and its required PostgreSQL cluster

FerretDB Cluster

Before deploying a Mongo (FerretDB) cluster, you need to define it in the inventory using the relevant parameters.

The following example uses the default single-node pg-meta cluster’s meta database as FerretDB’s underlying storage:

all:
  children:

    #----------------------------------#
    # ferretdb for mongodb on postgresql
    #----------------------------------#
    # ./mongo.yml -l ferret
    ferret:
      hosts:
        10.10.10.10: { mongo_seq: 1 }
      vars:
        mongo_cluster: ferret
        mongo_pgurl: 'postgres://mongod:[email protected]:5432/meta'

Here, mongo_cluster and mongo_seq are essential identity parameters. For FerretDB, mongo_pgurl is also required to specify the underlying PG location.

Note that the mongo_pgurl parameter requires a PostgreSQL superuser. In this example, a dedicated mongod superuser is defined for FerretDB.

Note that FerretDB’s authentication is entirely based on PostgreSQL. You can create other regular users using either FerretDB or PostgreSQL.

PostgreSQL Cluster

FerretDB 2.0+ requires an extension: DocumentDB, which depends on several other extensions. Here’s a template for creating a PostgreSQL cluster for FerretDB:

all:
  children:

    #----------------------------------#
    # pgsql (singleton on current node)
    #----------------------------------#
    # postgres cluster: pg-meta
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: mongod      ,password: DBUser.Mongo  ,pgbouncer: true ,roles: [dbrole_admin ] ,superuser: true ,comment: ferretdb super user }
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
        pg_databases:
          - {name: meta, owner: mongod ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ documentdb, postgis, vector, pg_cron, rum ]}
        pg_hba_rules:
          - { user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes' }
          - { user: mongod      , db: all ,addr: world ,auth: pwd ,title: 'mongodb password access from everywhere' }
        pg_extensions:
          - documentdb, citus, postgis, pgvector, pg_cron, rum
        pg_parameters:
          cron.database_name: meta
        pg_libs: 'pg_documentdb, pg_documentdb_core, pg_cron, pg_stat_statements, auto_explain'  # add timescaledb to shared_preload_libraries

High Availability

You can use Services to connect to a highly available PostgreSQL cluster and deploy multiple FerretDB instance replicas with L2 VIP binding for FerretDB layer high availability.

ferret:
  hosts:
    10.10.10.45: { mongo_seq: 1 }
    10.10.10.46: { mongo_seq: 2 }
    10.10.10.47: { mongo_seq: 3 }
  vars:
    mongo_cluster: ferret
    mongo_pgurl: 'postgres://mongod:[email protected]:5436/test'
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.99
    vip_interface: eth1

Parameters

There are 9 parameters in the MONGO module.

Parameter Type Level Comment
mongo_seq int I mongo instance identifier, REQUIRED
mongo_cluster string C mongo cluster name, MONGO by default
mongo_pgurl pgurl C/I underlying postgres URL for ferretdb
mongo_ssl_enabled bool C mongo/ferretdb ssl enabled, false by default
mongo_listen ip C mongo listen address, empty for all addr
mongo_port port C mongo service port, 27017 by default
mongo_ssl_port port C mongo tls listen port, 27018 by default
mongo_exporter_port port C mongo exporter port, 9216 by default
mongo_extra_vars string C extra environment variables for MONGO server 
# mongo_cluster:        #CLUSTER  # mongo cluster name, required identity parameter
# mongo_seq: 0          #INSTANCE # mongo instance seq number, required identity parameter
# mongo_pgurl: 'postgres:///'     # mongo/ferretdb underlying postgresql url, required
mongo_ssl_enabled: false          # mongo/ferretdb ssl enabled, false by default
mongo_listen: ''                  # mongo/ferretdb listen address, '' for all addr
mongo_port: 27017                 # mongo/ferretdb listen port, 27017 by default
mongo_ssl_port: 27018             # mongo/ferretdb tls listen port, 27018 by default
mongo_exporter_port: 9216         # mongo/ferretdb exporter port, 9216 by default
mongo_extra_vars: ''              # extra environment variables for mongo/ferretdb

3 - Administration

FerretDB administration playbooks, dashboards, and SOPs

Playbook

There’s a built-in playbook mongo.yml for installing the FerretDB cluster. But you have to define it first.

mongo.yml

mongo.yml: Install MongoDB/FerretDB on the target host.

This playbook consists of the following sub-tasks:

  • mongo_check : check mongo identity
  • mongo_dbsu : create os user mongod
  • mongo_install : install mongo/ferretdb rpm
  • mongo_purge : purge mongo/ferretdb
  • mongo_config : config mongo/ferretdb
    • mongo_cert : issue mongo/ferretdb ssl certs
  • mongo_launch : launch mongo/ferretdb service
  • mongo_register : register mongo/ferretdb to prometheus

Dashboard

There is one dashboard for FERRET module for now.

Mongo Overview

Mongo Overview: Overview of a Mongo/FerretDB cluster

mongo-overview.jpg

Create Mongo Cluster

After defining the MONGO cluster in the inventory, you can install it with:

./mongo.yml -l ferret   # Install MongoDB/FerretDB on ferret group

Since FerretDB uses PostgreSQL as its underlying storage, running this playbook multiple times is generally safe.

Remove Mongo Cluster

To remove a Mongo/FerretDB cluster, run the mongo_purge subtask of mongo.yml playbook with the mongo_purge parameter:

./mongo.yml -e mongo_purge=true -t mongo_purge

4 - Metrics

Pigsty MONGO module metric list

MONGO module has 54 available metrics

Metric Name Type Labels Description
ferretdb_client_accepts_total Unknown error, cls, ip, ins, instance, job N/A
ferretdb_client_duration_seconds_bucket Unknown error, le, cls, ip, ins, instance, job N/A
ferretdb_client_duration_seconds_count Unknown error, cls, ip, ins, instance, job N/A
ferretdb_client_duration_seconds_sum Unknown error, cls, ip, ins, instance, job N/A
ferretdb_client_requests_total Unknown cls, ip, ins, opcode, instance, command, job N/A
ferretdb_client_responses_total Unknown result, argument, cls, ip, ins, opcode, instance, command, job N/A
ferretdb_postgresql_metadata_databases gauge cls, ip, ins, instance, job The current number of database in the registry.
ferretdb_postgresql_pool_size gauge cls, ip, ins, instance, job The current number of pools.
ferretdb_up gauge cls, version, commit, ip, ins, dirty, telemetry, package, update_available, uuid, instance, job, branch, debug FerretDB instance state.
go_gc_duration_seconds summary cls, ip, ins, instance, quantile, job A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown cls, ip, ins, instance, job N/A
go_gc_duration_seconds_sum Unknown cls, ip, ins, instance, job N/A
go_goroutines gauge cls, ip, ins, instance, job Number of goroutines that currently exist.
go_info gauge cls, version, ip, ins, instance, job Information about the Go environment.
go_memstats_alloc_bytes gauge cls, ip, ins, instance, job Number of bytes allocated and still in use.
go_memstats_alloc_bytes_total counter cls, ip, ins, instance, job Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge cls, ip, ins, instance, job Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter cls, ip, ins, instance, job Total number of frees.
go_memstats_gc_sys_bytes gauge cls, ip, ins, instance, job Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge cls, ip, ins, instance, job Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge cls, ip, ins, instance, job Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge cls, ip, ins, instance, job Number of heap bytes that are in use.
go_memstats_heap_objects gauge cls, ip, ins, instance, job Number of allocated objects.
go_memstats_heap_released_bytes gauge cls, ip, ins, instance, job Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge cls, ip, ins, instance, job Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge cls, ip, ins, instance, job Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter cls, ip, ins, instance, job Total number of pointer lookups.
go_memstats_mallocs_total counter cls, ip, ins, instance, job Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge cls, ip, ins, instance, job Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge cls, ip, ins, instance, job Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge cls, ip, ins, instance, job Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge cls, ip, ins, instance, job Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge cls, ip, ins, instance, job Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge cls, ip, ins, instance, job Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge cls, ip, ins, instance, job Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge cls, ip, ins, instance, job Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge cls, ip, ins, instance, job Number of bytes obtained from system.
go_threads gauge cls, ip, ins, instance, job Number of OS threads created.
mongo_up Unknown cls, ip, ins, instance, job N/A
process_cpu_seconds_total counter cls, ip, ins, instance, job Total user and system CPU time spent in seconds.
process_max_fds gauge cls, ip, ins, instance, job Maximum number of open file descriptors.
process_open_fds gauge cls, ip, ins, instance, job Number of open file descriptors.
process_resident_memory_bytes gauge cls, ip, ins, instance, job Resident memory size in bytes.
process_start_time_seconds gauge cls, ip, ins, instance, job Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge cls, ip, ins, instance, job Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge cls, ip, ins, instance, job Maximum amount of virtual memory available in bytes.
promhttp_metric_handler_errors_total counter job, cls, ip, ins, instance, cause Total number of internal errors encountered by the promhttp metric handler.
promhttp_metric_handler_requests_in_flight gauge cls, ip, ins, instance, job Current number of scrapes being served.
promhttp_metric_handler_requests_total counter job, cls, ip, ins, instance, code Total number of scrapes by HTTP status code.
scrape_duration_seconds Unknown cls, ip, ins, instance, job N/A
scrape_samples_post_metric_relabeling Unknown cls, ip, ins, instance, job N/A
scrape_samples_scraped Unknown cls, ip, ins, instance, job N/A
scrape_series_added Unknown cls, ip, ins, instance, job N/A
up Unknown cls, ip, ins, instance, job N/A

5 - FAQ

Pigsty FerretDB / MONGO module frequently asked questions

Install mongosh

cat > /etc/yum.repos.d/mongo.repo <<EOF
[mongodb-org-7.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/7.0/x86_64/
gpgcheck=0
enabled=1
EOF

yum install -y mongodb-mongosh

# or just install via rpm & links
rpm -ivh https://mirrors.tuna.tsinghua.edu.cn/mongodb/yum/el7/RPMS/mongodb-mongosh-1.9.1.x86_64.rpm