Unable to ignore properties when deserializing. TokenFilter seems broken

[xiazuojie]

We have the following JSON generated by another JSON library that includes property @type to indicate value types. We want Jackson to ignore @type properties (both root and inner ones) when deserializing the data to Map<Long, Map<Long, Long>>.

{
   "@type":"java.util.LinkedHashMap",
   "111":{
      "@type":"java.util.LinkedHashMap",
      "1":11,
      "2":22
   }
}

What we have tried so far:

• ObjectReader.withoutAttribute("@type")
• ObjectMapper.addMixIn
• TokenFilter
• upgrade Jackson to latest version 2.12.3

None of the above gets what we want.
The closest result we've got is by using TokenFilter. However, there's always something broker no matter what combination of TokenFilter options we choose:

• TokenFilter#includeProperty returning this or TokenFilter#INCLUDE_ALL
• all TokenFilter.Inclusion enums for FilteringParserDelegate#_inclusion
• FilteringParserDelegate#_allowMultipleMatches true or false

It seems TokenFilter is broken. The following is our test case.

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.filter.FilteringParserDelegate;
import com.fasterxml.jackson.core.filter.TokenFilter;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ObjectReader;
import org.junit.Test;

import java.util.Map;

import static org.junit.Assert.assertEquals;

public class IgnorePropertyTest {
    private final ObjectMapper mapper = new ObjectMapper();
    private final String LONG_INNER_MAP_WITH_AUTO_TYPE = "{\"@type\":\"java.util.LinkedHashMap\",\"111\":{\"@type\":\"java.util.LinkedHashMap\",\"1\":11,\"2\":22}}";

    @Test
    public void testTokenFilter() throws Exception {
        final TokenFilter tokenFilter = new TokenFilter() {
            @Override
            public TokenFilter includeProperty(String name) {
                if ("@type".equals(name)) {
                    return null;
                }
                return this;
            }
        };
        JsonParser jsonParser = new FilteringParserDelegate(
                mapper.createParser(LONG_INNER_MAP_WITH_AUTO_TYPE),
                tokenFilter,
                TokenFilter.Inclusion.INCLUDE_ALL_AND_PATH,
                true);
        Map<Long, Map<Long, Long>> data = mapper.readValue(jsonParser, new TypeReference<Map<Long, Map<Long, Long>>>() {});
        check(data);
    }

    @Test
    public void testReader() throws Exception {
        ObjectReader reader = mapper.readerFor(new TypeReference<Map<Long, Map<Long, Long>>>() {});
        Map<Long, Map<Long, Long>> data = reader.withoutAttribute("@type").readValue(LONG_INNER_MAP_WITH_AUTO_TYPE);
        check(data);
    }

    @Test
    public void testMixin() throws Exception {
        ObjectMapper mp = mapper.addMixIn(Map.class, MixIn.class);
        Map<Long, Map<Long, Long>> data = mp.readValue(LONG_INNER_MAP_WITH_AUTO_TYPE, new TypeReference<Map<Long, Map<Long, Long>>>() {});
        check(data);
    }

    @JsonIgnoreProperties({"@kind"})
    private static abstract class MixIn {
    }

    private void check(Map<Long, Map<Long, Long>> data) {
        assertEquals(1, data.size());
        assertEquals(2, data.get(111L).size());
        assertEquals(11L, (long)data.get(111L).get(1L));
        assertEquals(22L, (long)data.get(111L).get(2L));
    }
}

Jackson version: 2.12.3

[cowtowncoder]

First things first: withoutAttribute() would not have any effect as "attribute" is metadata associated with DeserializationContext and has no direct effect on input being read. Entries in JSON Objects are never called attributes in Jackson API (they are referred to as "properties" or in some cases "fields").

Mix-in, however, should safely ignore field. But your test case above has 2 problems:

1. Mix-in adds ignore for @kind (@JsonIgnoreProperties({"@kind"})) -- but you are feeding @type?
2. Code is trying to register mix-in dynamically but it MUST be registered RIGHT AFTER construction: the first time ObjectMapper is used it will use current settings and further changes will have no effect. In this case it may or may not be ignored, depending on order in which tests are run. So you should create separate ObjectMapper for test cases

So I think it should be possible to make this (mix-in) approach work.

Filter case is interesting and I don't know for sure what the issue might be: I would like to have a look if I have time, to see what is causing issues.

Also, if you can specify a helper class like:

@JsonIgnoreProperties({"@type"})
public class MyLongMap extends
    HashMap<Long, Map<Long, Long>> {
}

you can avoid using mix-ins, and simplify code to something like

MyLongMap  map = mapper.readValue(json, MyLongMap.class);

which might be the simplest way to do it.

[xiazuojie]

hi, @cowtowncoder, I've updated the test case.
Unfortunately, none of the methods you mentioned solved the problem.

package com.alibaba.dapr.serialization.json;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.filter.FilteringParserDelegate;
import com.fasterxml.jackson.core.filter.TokenFilter;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ObjectReader;
import org.junit.Ignore;
import org.junit.Test;

import java.util.HashMap;
import java.util.Map;

import static org.junit.Assert.assertEquals;

@Ignore
public class IgnorePropertyTest {
    private final ObjectMapper mapper = new ObjectMapper().addMixIn(Map.class, MixIn.class);
    private final String LONG_INNER_MAP_WITH_AUTO_TYPE = "{\"@type\":\"java.util.LinkedHashMap\",\"111\":{\"@type\":\"java.util.LinkedHashMap\",\"1\":11,\"2\":22}}";

    @Test
    public void testTokenFilter() throws Exception {
        final TokenFilter tokenFilter = new TokenFilter() {
            @Override
            public TokenFilter includeProperty(String name) {
                if ("@type".equals(name)) {
                    return null;
                }
                return this;
            }
        };
        JsonParser jsonParser = new FilteringParserDelegate(
                mapper.createParser(LONG_INNER_MAP_WITH_AUTO_TYPE),
                tokenFilter,
                TokenFilter.Inclusion.INCLUDE_ALL_AND_PATH,
                true);
        Map<Long, Map<Long, Long>> data = mapper.readValue(jsonParser, new TypeReference<Map<Long, Map<Long, Long>>>() {});
        check(data);
    }

    @Test
    public void testReader() throws Exception {
        ObjectReader reader = mapper.readerFor(new TypeReference<Map<Long, Map<Long, Long>>>() {});
        Map<Long, Map<Long, Long>> data = reader.withoutAttribute("@type").readValue(LONG_INNER_MAP_WITH_AUTO_TYPE);
        check(data);
    }

    @Test
    public void testMixin() throws Exception {
        Map<Long, Map<Long, Long>> data = mapper.readValue(LONG_INNER_MAP_WITH_AUTO_TYPE, new TypeReference<Map<Long, Map<Long, Long>>>() {});
        check(data);
    }

    @Test
    public void testCustomMap() throws Exception {
        Map data = mapper.readValue(LONG_INNER_MAP_WITH_AUTO_TYPE, new TypeReference<MyLongMap>() {});
        check(data);
    }

    @JsonIgnoreProperties({"@type"})
    public static class MyLongMap extends HashMap<Long, MyInnerLongMap> {
    }

    @JsonIgnoreProperties({"@type"})
    public static class MyInnerLongMap extends HashMap<Long, Long> {
    }

    @JsonIgnoreProperties({"@type"})
    private static abstract class MixIn {
    }

    private void check(Map<Long, Map<Long, Long>> data) {
        assertEquals(1, data.size());
        assertEquals(2, data.get(111L).size());
        assertEquals(11L, (long)data.get(111L).get(1L));
        assertEquals(22L, (long)data.get(111L).get(2L));
    }
}

[xiazuojie]

I've achieved my goal by writing my own filtering parser.

package com.alibaba.dapr.serialization.json;

import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.core.util.JsonParserDelegate;

import java.io.IOException;

import static com.fasterxml.jackson.core.JsonTokenId.ID_FIELD_NAME;

/**
 * Ignores com.alibaba.fastjson.JSON#DEFAULT_TYPE_KEY.
 * Because {@link com.fasterxml.jackson.core.filter.FilteringParserDelegate} is broken at the moment,
 * we have to write our own simple filtering parser.
 * @author zuojie
 */
public class TypeFieldFilteringParserDelegate extends JsonParserDelegate {
    public TypeFieldFilteringParserDelegate(JsonParser d) {
        super(d);
    }

    public JsonToken nextToken() throws IOException {
        JsonToken t = delegate.nextToken();
        if (t == null) {
            return null;
        }
        if (t.id() == ID_FIELD_NAME) {
            final String name = delegate.getCurrentName();
            if ("@type".equals(name)) {
                delegate.nextToken();
                delegate.skipChildren();
                return delegate.nextToken();
            }
        }
        return t;
    }
}

package com.alibaba.dapr.serialization.json;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.filter.FilteringParserDelegate;
import com.fasterxml.jackson.core.filter.TokenFilter;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ObjectReader;
import org.junit.Ignore;
import org.junit.Test;

import java.util.HashMap;
import java.util.Map;

import static org.junit.Assert.assertEquals;

@Ignore
public class IgnorePropertyTest {
    private final ObjectMapper mapper = new ObjectMapper().addMixIn(Map.class, MixIn.class);
    private final String LONG_INNER_MAP_WITH_AUTO_TYPE = "{\"@type\":\"java.util.LinkedHashMap\",\"111\":{\"@type\":\"java.util.LinkedHashMap\",\"1\":11,\"2\":22}}";

    @Test
    public void testTokenFilter() throws Exception {
        final TokenFilter tokenFilter = new TokenFilter() {
            @Override
            public TokenFilter includeProperty(String name) {
                if ("@type".equals(name)) {
                    return null;
                }
                return this;
            }
        };
        JsonParser jsonParser = new FilteringParserDelegate(
                mapper.createParser(LONG_INNER_MAP_WITH_AUTO_TYPE),
                tokenFilter,
                TokenFilter.Inclusion.INCLUDE_ALL_AND_PATH,
                true);
        Map<Long, Map<Long, Long>> data = mapper.readValue(jsonParser, new TypeReference<Map<Long, Map<Long, Long>>>() {});
        check(data);
    }

    @Test
    public void testReader() throws Exception {
        ObjectReader reader = mapper.readerFor(new TypeReference<Map<Long, Map<Long, Long>>>() {});
        Map<Long, Map<Long, Long>> data = reader.withoutAttribute("@type").readValue(LONG_INNER_MAP_WITH_AUTO_TYPE);
        check(data);
    }

    @Test
    public void testMixin() throws Exception {
        Map<Long, Map<Long, Long>> data = mapper.readValue(LONG_INNER_MAP_WITH_AUTO_TYPE, new TypeReference<Map<Long, Map<Long, Long>>>() {});
        check(data);
    }

    @Test
    public void testCustomMap() throws Exception {
        Map data = mapper.readValue(LONG_INNER_MAP_WITH_AUTO_TYPE, new TypeReference<MyLongMap>() {});
        check(data);
    }

    @Test
    public void testCustomFilteringParserDelegate() throws Exception {
        JsonParser jsonParser = new TypeFieldFilteringParserDelegate(mapper.createParser(LONG_INNER_MAP_WITH_AUTO_TYPE));
        Map data = mapper.readValue(jsonParser, new TypeReference<MyLongMap>() {});
        check(data);
    }

    @JsonIgnoreProperties({"@type"})
    public static class MyLongMap extends HashMap<Long, MyInnerLongMap> {
    }

    @JsonIgnoreProperties({"@type"})
    public static class MyInnerLongMap extends HashMap<Long, Long> {
    }

    @JsonIgnoreProperties({"@type"})
    private static abstract class MixIn {
    }

    private void check(Map<Long, Map<Long, Long>> data) {
        assertEquals(1, data.size());
        assertEquals(2, data.get(111L).size());
        assertEquals(11L, (long)data.get(111L).get(1L));
        assertEquals(22L, (long)data.get(111L).get(2L));
    }
}

testCustomFilteringParserDelegate finally works.

[cowtowncoder]

@xiazuojie thank you for sharing your solution. I hope to have a look at your original case to see why ignoral does not work, but at least you have a work-around.

[cowtowncoder]

I am able to reproduce this with even simpler case:

{'@type':'...','value':12}

which produces erroneous stream of

{ "12" }

(that is, missing value).

[cowtowncoder]

Phew! Was able to figure out issues: turns out there was a code path commented out (for reasons unknown to me) responsible for the main issue -- but also, at databind level there was a minor quirk in Map deserializer as well.

Fix will be in 2.13.0-rc2 (if one published) or final 2.13.0 (if no more release candidates).

Thank you @xiazuojie for reporting this problem!