Querying data by using DataPrime in IBM Cloud Logs

In IBM Cloud Logs, you can search log data by using DataPrime. Dataprime is an advanced query language that filters, transforms, and aggregates data by using commands that are combined by using the pipe (|) operator and applied sequentially.

Use Dataprime queries to filter, transform, and aggregate data by using commands that help you manipulate log records with a language that offers full expression support similar to programming languages.

Searching can be used in conjunction with filtering and limiting by time.

Building a Dataprime query in IBM Cloud Logs

You can define a Dataprime query using the following format:

source logs | operator1 ... | operator2 ... | operator3 | ...

Any whitespace between operators is ignored.

A pipe character (|) is used to chain sequential operations, passing transformed or filtered data from one step to the next in a pipeline.

You can also define Dataprime queries across multiple lines:

source logs |
operator1 expression |
operator2 expression |
operator3 expression

Expressions such as filter status_code == 403 compute or compare any value. Every expression must return a result, like a string, number, boolean, timestamp, or array.

Operators are commands that transform unstructured streams of JSON data to create a new stream for a specified time period. Examples include: filter, extract, sortby, groupby, orderby, find, and choose. For more information about DataPrime operators, see the DataPrime operators.

Functions in DataPrime queries transform, compute, and manipulate telemetry data at the field level. For more information, see:

Commands must be in the correct order. For example, the create command must be set before the filter command if you're filtering on a derived field.

Use a command when transforming the shape, size, or structure of your dataset.

Use a function when transforming, computing or manipulating individual field values.

Type mismatches can break filters or aggregations. Use the convert command or inline casting to set the expected type. Inline casting is performed using a colon suffix syntax (:type) directly on fields or expressions, for example, filter status_code:number > 403.

Running a Dataprime query in the Logs page in IBM Cloud Logs

You can run a Dataprime query in the Explore Logs > Logs page.

Complete the following steps to run a query in the Logs page in IBM Cloud Logs:

  1. Launch the IBM Cloud Logs UI.

  2. Click the Explore logs icon icon Explore logs icon. Then select Logs.

    By default, Priority Insights is selected.

    If you want to explore data from the COS bucket, click All Logs.

  3. In the query section, select </> DataPrime to specify your search to be a DataPrime query.

    By default, the search query mode is set to Lucene.

  4. Enter a Dataprime query. Then, click the Play icon to run your query.

    For more information, see Building a Dataprime query in IBM Cloud Logs.

If you are searching in Priority insights and you are not seeing the data you expect, see Why am I not seeing results from my Priority insights searches?.

Running a Dataprime query in a custom dashboard widget in IBM Cloud Logs

You can run a Dataprime query in the Dashboards > Custom Dashboards page.

Complete the following steps to run a query in the Custom Dashboards page in IBM Cloud Logs:

  1. Launch the IBM Cloud Logs UI.

  2. Click the Dashboards icon icon Dashboards icon. Then select Custom Dashboards.

  3. Create a new dashboard and add a widget. You can also add a widget to an existing dashboard.

  4. In the widget's configuration panel, select Dataprime.

    Dataprime query in a custom dashboard

  5. Enter a Dataprime query.

    For more information, see Building a Dataprime query in IBM Cloud Logs.

If you are searching in Priority insights and you are not seeing the data you expect, see Why am I not seeing results from my Priority insights searches?.