Object Filter

A json structure to express filters.

The most basic filter

The most basic filter indicates with key which property of the object to read, and contains a list of values indicating which values will result in a match.

filter
{
    "key": "/annotation/funnel",
    "values": ["ClipAnnotation", "ReviewComment"]
}

The special value "." for key means use the object itsself for matching.

The following object would match the filter

object to match against
{
    "annotation": {
        "funnel": "ReviewComment"
    }
}

Valid values for key

key can be specified as a Json Pointer, or as "dot syntax".

When the key does not start with /, it is assumed to use dot syntax.

  • /someObject/deepValue in dot syntax is someObject.deepValue

  • /someArray/0/deepValue in dot syntax is someArray[0].deepValue

  • / in dot syntax is .

Usage of Object Filters in some places also supports using FlowJsonPointer, which is an extension of JsonPointer.

Matching null, undefined and missing properties

The difference between undefined and null is javascript-specific. Object Filter implementations in other programming languages might treat this slightly different.

  • Use undefinedMatches to match against undefined. This means the object explicitly has undefined as value for the key, or that the key is missing.

  • Use nullMatches to match against null. This is equivalent to adding null to the values array. It means the object explicitly has null as value for the key.

  • Use missingMatches as a shorthand which means both undefinedMatches and nullMatches are true.

filter with missingMatches
{
    "key": "annotation.funnel",
    "values": ["ClipAnnotation", "ReviewComment"],
    "missingMatches": true
}

NOT

What if we wanted our filter to match everything except for the given values?

Easy! Just add a NOT modifier as shown below:

filter with NOT
{
    "key": "annotation.funnel",
    "values": ["ClipAnnotation", "ReviewComment"],
    "modifiers": ["NOT"]
}

Case Insensitive Matching

Another useful modifier is CASE_INSENSITIVE. When comparing strings, it will ignore differences in casing. When using operator "REGEX", it will pass the i flag.

Combining filters

Now we’ll look at combining filters.

This is done using an object with a list of filters and an operator (which defaults to AND).

  • When the operator is "OR", the filter matches when any of the filters in the list match.

  • When the operator is "AND" (or omitted), the filter matches when all of the filters in the list match.

combination using OR
{
    "operator": "OR",
    "filters": [
        {
            "key": "annotation.funnel",
            "values": ["ClipAnnotation", "ReviewComment"]
        },
        {
            "key": "annotation.description",
            "values": "a desc"
        },
        {
            "key": "annotation.description",
            "values": "forbidden desc",
            "modifiers": ["NOT"]
        }
    ]
}

Note that each filter in the filters list can again be a combination of filters, so boolean logic can be expressed with this.

An array should contain something

Say key points to an array and we want to check if a value is present in this array.

In that case, one can use the ARRAY_CONTAINS_ALL and ARRAY_CONTAINS_ANY operators.

{
    "key": "userRightsArray",
    "values": ["LIBRARY_UPLOAD","LIBRARY_DELETE"],
    "operator": "ARRAY_CONTAINS_ALL"
}

The following object would not match, because object.userRightsArray does not include LIBRARY_DELETE

{
  "userRightsArray": ["PRODUCTION_VIEW", "LIBRARY_UPLOAD"]
}

  • When the operator is "ARRAY_CONTAINS_ANY", the array (pointed to by key) should contain at least one of the values.

    • If undefinedMatches is true, an undefined value in the array will make the filter match.

    • If nullMatches is true, a null value in the array will make the filter match.

    • If missingMatches is true, a null or undefined value in the array will make the filter match.

  • When the operator is "ARRAY_CONTAINS_ALL", the array (pointed to by key') should contain all the `values.

    • If undefinedMatches is true, the array should also contain undefined.

    • If nullMatches is true, the array should also contain null.

    • If missingMatches is true, the array should also contain both null and undefined (!).

An array should contain an element which matches something

Say key points to an array with objects. We want to check if an object in the array exists which matches a number of filters.

In that case, one can use the ARRAY_ELEMENT_MATCHES_ALL and ARRAY_ELEMENT_MATCHES_ANY operators.

When the operator is one of these operators, the 'object' in its filters refers to a single object in the array. One can then again use the full range of filters to match an object in the array. If (and only if) an object is found which matches, the ARRAY_ELEMENT_MATCHES_* filter evaluates to true.

  • When the operator is "ARRAY_ELEMENT_MATCHES_ALL", an object should be found in the array for which all the filters match.

  • When the operator is "ARRAY_ELEMENT_MATCHES_ANY", an object should be found in the array for which at least one of the filters matches.

{
    "key": "volumeLocation",
    "operator": "ARRAY_ELEMENT_MATCHES_ALL",
    "filters": [
        {
            "key": "volume.handle",
            "values": ["flow-nearline"]
        },
        {
            "key": "shouldBeOnVolume",
            "values": [false]
        },
        {
            "key": "onVolume",
            "values": [true]
        }
    ]
}

The following object matches:

{
    "volumeLocation": [
        {
            "volume": {
                "handle": "flow-nearline"
            },
            "shouldBeOnVolume": false,
            "onVolume": true
        }
    ]
}

The following object does not match:

{
    "volumeLocation": [
        {
            "volume": {
                "handle": "different"
            },
            "shouldBeOnVolume": false,
            "onVolume": true
        }
    ]
}

Range Filters

The IN_RANGE operator can be used to check if a numeric value falls within a range.

{
    "key": "annotation.start",
    "operator": "IN_RANGE",
    "range": {
        "start": 200,
        "end": 300
    }
}

The following object matches, as 250 is in [200,300]

{
    "annotation": {
        "start": 250
    }
}
Property Description

start

Start of the range. A number or the String "" to indicate unbounded. When omitted, "" is used.

end

End of the range. A number or the String "" to indicate unbounded. When omitted, "" is used.

startInclusive

When the value equals the start of the range, the filter will match if and only if startInclusive is true. The default is true (start inclusive).

endInclusive

When the value equals the end of the range, the filter will match if and only if endInclusive is true. The default is true (end inclusive).

Date Range Filters

The IN_DATE_RANGE operator can be used to check if a date value falls within a range.

It works the same as the IN_RANGE operator, but the start and end will be Date expressions.

The Date expressions are based on Solr Date Math Syntax: https://solr.apache.org/guide/solr/latest/indexing-guide/date-formatting-math.html#date-math-syntax.

For example, start could be "*" and end could be "NOW/MONTH+7DAYS-1HOURS" to indicate a date up to the current date, rounded up to the month, to which we add 7 days and subtract 1 hour.

Besides start, end, startInclusive and endInclusive, there is one additional option that can be set: the now option. This can be used to force NOW to a fixed date. It should be specified as an ISO 8601 date-time string.

{
    "key": "annotation.created",
    "operator": "IN_DATE_RANGE",
    "range": {
        "start": "*",
        "end": "NOW-1HOURS",
        "now": "2024-03-07T01:02:03.040Z"
    }
}

The following object matches:

{
    "annotation": {
        "created": 1709773323040 // this is new Date('2024-03-07T01:02:03.040Z').getTime()
    }
}

The Date Math expressions can also contain ISO 8601 date-time string directly instead of NOW.

Regular Expression Filter

Advanced string matching can be done using regular expresions.

Set operator to "REGEX" and put the regex in pattern:

{
    "key": "volumeHandle",
    "operator": "REGEX",
    "pattern": "^flow.*"
}

The object { volumeHandle: "flow_123" } matches this filter.