28.4.6. EVE Filetypes

28.4.6.1. Introduction

The Suricata EVE/JSON output supports filetypes to extend how EVE records are processed and delivered. Custom filetypes provide alternatives to standard file output by implementing a file-like interface that Suricata can write to. These filetypes can send events to databases, sockets, or other destinations, and can even perform custom processing on the output before storing it.

28.4.6.2. EVE Filetype Life Cycle

The life-cycle of an EVE filetype along with the callbacks are discussed in output-eve.h:

/** \brief Structure used to define an EVE output file type.
 *
 * EVE filetypes implement an object with a file-like interface and
 * are used to output EVE log records to files, syslog, or
 * database. They can be built-in such as the syslog (see
 * SyslogInitialize()) and nullsink (see NullLogInitialize()) outputs,
 * registered by a library user or dynamically loaded as a plugin.
 *
 * The life cycle of an EVE filetype is:
 *   - Init: called once for each EVE instance using this filetype
 *   - ThreadInit: called once for each output thread
 *   - Write: called for each log record
 *   - ThreadDeinit: called once for each output thread on exit
 *   - Deinit: called once for each EVE instance using this filetype on exit
 *
 * Examples:
 * - built-in syslog: \ref src/output-eve-syslog.c
 * - built-in nullsink: \ref src/output-eve-null.c
 * - example plugin: \ref examples/plugins/c-json-filetype/filetype.c
 *
 * ### Multi-Threaded Note:
 *
 * The EVE logging system can be configured by the Suricata user to
 * run in threaded or non-threaded modes. In the default non-threaded
 * mode, ThreadInit will only be called once and the filetype does not
 * need to be concerned with threads.
 *
 * However, in **threaded** mode, ThreadInit will be called multiple
 * times and the filetype needs to be thread aware and thread-safe. If
 * utilizing a unique resource such as a file for each thread then you
 * may be naturally thread safe. However, if sharing a single file
 * handle across all threads then your filetype will have to take care
 * of locking, etc.
 */
typedef struct SCEveFileType_ {
    /**
     * \brief The name of the output, used in the configuration.
     *
     * This name is used by the configuration file to specify the EVE
     * filetype used.
     *
     * For example:
     *
     * \code{.yaml}
     * outputs:
     *   - eve-log:
     *       filetype: my-output-name
     * \endcode
     */
    const char *name;

    /**
     * \brief Function to initialize this filetype.
     *
     * \param conf The ConfNode of the `eve-log` configuration
     *     section this filetype is being initialized for
     *
     * \param threaded Flag to specify if the EVE sub-systems is in
     *     threaded mode or not
     *
     * \param init_data An output pointer for filetype specific data
     *
     * \retval 0 on success, -1 on failure
     */
    int (*Init)(const SCConfNode *conf, const bool threaded, void **init_data);

    /**
     * \brief Initialize thread specific data.
     *
     * Initialize any thread specific data. For example, if
     * implementing a file output you might open the files here, so
     * you have one output file per thread.
     *
     * \param init_data Data setup during Init
     *
     * \param thread_id A unique ID to differentiate this thread from
     *     others. If EVE is not in threaded mode this will be called
     *     once with a ThreadId of 0. In threaded mode the ThreadId of
     *     0 correlates to the main Suricata thread.
     *
     * \param thread_data Output pointer for any data required by this
     *     thread.
     *
     * \retval 0 on success, -1 on failure
     */
    int (*ThreadInit)(const void *init_data, const ThreadId thread_id, void **thread_data);

    /**
     * \brief Called for each EVE log record.
     *
     * The Write function is called for each log EVE log record. The
     * provided buffer contains a fully formatted EVE record in JSON
     * format.
     *
     * \param buffer The fully formatted JSON EVE log record
     *
     * \param buffer_len The length of the buffer
     *
     * \param init_data The data setup in the call to Init
     *
     * \param thread_data The data setup in the call to ThreadInit
     *
     * \retval 0 on success, -1 on failure
     */
    int (*Write)(
            const char *buffer, const int buffer_len, const void *init_data, void *thread_data);

    /**
     * \brief Called to deinitialize each thread.
     *
     * This function will be called for each thread. It is where any
     * resources allocated in ThreadInit should be released.
     *
     * \param init_data The data setup in Init
     *
     * \param thread_data The data setup in ThreadInit
     */
    void (*ThreadDeinit)(const void *init_data, void *thread_data);

    /**
     * \brief Final call to deinitialize this filetype.
     *
     * Called, usually on exit to deinitialize and free any resources
     * allocated during Init.
     *
     * \param init_data Data setup in the call to Init.
     */
    void (*Deinit)(void *init_data);

    /* Internal list management. */
    TAILQ_ENTRY(SCEveFileType_) entries;
} SCEveFileType;

28.4.6.3. Threading Considerations

It is the user's Suricata EVE output configuration that enables multi-threaded logging, not the filetype. So all filetypes should be designed to be thread safe.

If your filetype can absolutely not be made thread safe, it would be best to error out on initialization. This can be done during the filetype initialization:

static int MyFiletypeInit(const SCConfNode *node, const bool threaded, void **data)
{
    if (threaded) {
        FatalError("EVE filetype does not support threaded logging.");
    }

    /* Continue with initialization. */
}

28.4.6.4. Write Considerations

The Write callback is called in a packet processing thread so any blocking (other than writing to a file) should be avoided. If writing to a blocking resource it is recommended to copy the buffer into another thread for further processing to avoid packet loss.

28.4.6.5. Registration

Registering an EVE filetype requires registering the filetype early in the Suricata start-up or lifecycle, or if a plugin, in the plugin initialization function.

SCEveFileType *filetype = SCCalloc(1, sizeof(SCEveFileType));

filetype->name = "my-custom-filetype";
filetype->Init = FiletypeInit;
filetype->Deinit = FiletypeDeinit;
filetype->ThreadInit = FiletypeThreadInit;
filetype->ThreadDeinit = FiletypeThreadDeinit;
filetype->Write = FiletypeWrite;

if (!SCRegisterEveFileType(filetype)) {
    FatalError("Failed to register EVE filetype");
}

Then to use this filetype, set the filetype in your suricata.yaml eve-log configuration to the name of the filetype:

outputs:
  - eve-log:
      enabled: true
      filetype: my-custom-filetype

28.4.6.6. Examples

Suricata built-ins:

• null: see output-eve-null.c in the Suricata source code

• syslog: see output-eve-syslog.c in the Suricata source code

Plugin:

• The Suricata source code contains an example as a plugin, see: examples/plugins/c-json-filetype.