Null-Checker gives false positive when calling library code

Is there a way to view the reasoning of IntelliJ's analyser?

I suspect that the problem is that the Function's type parameters lose their annotations during compilation:

Function<? super T, U>

Maybe IntelliJ then interprets that to mean "no idea if those are nullable or not, better safe than sorry!". Is it possible to declare a Function parameter in a way that tells IntelliJ that the input of the Function will never be null and that also survives compilation into a library?

Why does it suspect nullability anyway? Shouldn't non-annotated code be treated as "I trust you" anyway? 

From the description it looks like the issue might be in type inference, which infers the function parameter to be of "@Nullable String" type. But to check that, I'd need to reproduce it. Do you have compiled jar somewhere?

Yes, the library is on Bintray/JCenter (I can't link to the jar directly, but you can find it in the "Files" tab): https://bintray.com/cosee/null4j/null4j#files/biz%2Fcosee%2Fnull4j%2Fnull4j%2F0.9.0

You can use it like this:

<repository>
    <id>jcenter</id>
    <url>http://jcenter.bintray.com </url>
</repository>

<dependency>
    <groupId>biz.cosee.null4j</groupId>
    <artifactId>null4j</artifactId>
    <version>0.9.0</version>
</dependency>

I was wondering, why does the JetBrains Nullable annotation not target ElementType.TYPE_PARAMETER? The IntelliJ analyser clearly takes type parameter annotations into account.

Even if I copy the Nullable annotation, add ElementType.TYPE_PARAMETER to the targets and set the Retention to RetentionPolicy.RUNTIME it still won't show up in the decompiled code (also, copying the annotation isn't an option anyway since IntelliJ will just ignore it if it's not the real JetBrains annotation).

Any idea how to get the annotations to stick? If it's not possible to keep type parameter annotations in a compiled library, is there another way to assure the IntelliJ checker that the Function passed as a parameter to "let" will never have to deal with null input and may output null if it wants to?

Okay, I think I found a problem with IntelliJ's decompiler.

You know how it's weird that the annotation doesn't show up when you open the .class file in IntelliJ with the fernflower compiler? Well, the annotation is actually there if you disassemble it with javap:

public static <T extends java.lang.Object, U extends java.lang.Object> U let(T, java.util.function.Function<? super T, U>);
descriptor: (Ljava/lang/Object;Ljava/util/function/Function;)Ljava/lang/Object;
flags: ACC_PUBLIC, ACC_STATIC
Code:
  stack=2, locals=2, args_size=2
  StackMap locals:  java/lang/Object java/util/function/Function
  StackMap stack:
    start local 0 // java.lang.Object t
    start local 1 // java.util.function.Function f_T_U
     0: aload_0
     1: ifnonnull     6
     4: aconst_null
     5: areturn
  StackMap locals:  java/lang/Object java/util/function/Function
  StackMap stack:
     6: aload_1
     7: aload_0
     8: invokeinterface #2,  2            // InterfaceMethod java/util/function/Function.apply:(Ljava/lang/Object;)Ljava/lang/Object;
    13: areturn
    end local 1 // java.util.function.Function f_T_U
    end local 0 // java.lang.Object t
  LineNumberTable:
    line 276: 0
    line 277: 6
  LocalVariableTable:
    Start  Length  Slot  Name   Signature
        0      14     0     t   Ljava/lang/Object;
        0      14     1 f_T_U   Ljava/util/function/Function;
  LocalVariableTypeTable:
    Start  Length  Slot  Name   Signature
        0      14     0     t   TT;
        0      14     1 f_T_U   Ljava/util/function/Function<-TT;TU;>;
  StackMapTable: number_of_entries = 1
    frame_type = 6 /* same */
Signature: #57                          // <T:Ljava/lang/Object;U:Ljava/lang/Object;>(TT;Ljava/util/function/Function<-TT;TU;>;)TU;
RuntimeInvisibleAnnotations:
  0: #25()
RuntimeInvisibleTypeAnnotations:
  0: #25(): METHOD_RETURN
  1: #25(): METHOD_FORMAL_PARAMETER, param_index=0
  2: #25(): METHOD_FORMAL_PARAMETER, param_index=1, location=[TYPE_ARGUMENT(1)]
RuntimeInvisibleParameterAnnotations:
  0:
    0: #25()
  1:

So the IntelliJ decompiler somehow forgets about that @Nullable annotation... is this compiler output used by IntelliJ's nullchecker? Could this explain the issue and can this be fixed?

Which version of IntelliJ IDEA are you seeing this issue in? I see no yellow code in your code sample with 2017.2 EAPs, and no difference with source-based "let" either.

ElementType.TYPE_PARAMETER means type parameter declaration, not type argument (as String is in List<String>). class Foo<@Nullable T> doesn't seem to make much sense, so we don't allow that.

Indeed, some type annotations are not yet correctly parsed from class files by the IDE. In this particular case, only the second argument of Function seems to be affected, and this shouldn't influence the highlighting.

Hi Peter,

I'm using IntelliJ 2017.1.4 from June 6 (I just upgraded it) but the problem also happened in the May version.

I configured the @NotNull/@Nullable related problems and the Constant Conditions problems as Errors, maybe that makes a difference.

I also isolated the problem further, the null checker behaviour does not make much sense in this case.

In this case, the ".replace" gets underlined because the checker somehow believes that "string" is @Nullable:

private @Nullable String replaceSpecialChars(@Nullable String str) {
    return let(str, string -> {
        return orDefault(string.replace('/', '_'),
                "wtf");
    });
}

This is somehow fine (notice that "str" is no longer @Nullable):

private @Nullable String replaceSpecialChars(String str) {
    return let(str, string -> {
        return orDefault(string.replace('/', '_'),
                "wtf");
    });
}

This makes no sense since the nullability of the first parameter of let has no impact on the nullability of the first parameter of the Function that is passed to let. In the old implementation this was very obvious:

public static <T, U>
@Nullable U let(
        @Nullable T t,
        Function<? super T, U> f_T_U
) {
    if(t == null) return null;
    return f_T_U.apply(t);
}

I have changed the implementation to something less obvious but more assertive but the problem still remains:

public static <T, U>
@Nullable U let(
        @Nullable T t,
        Function<? super T, U> f_T_U
) {
    if(t == null) return null;
    @NotNull T nn_t = surely(t);
    return f_T_U.apply(nn_t);
}

private static <T>
@NotNull T surely(@Nullable T t) {
    Supplier<@NotNull T> hack = () -> (@NotNull T) Function.identity().apply(t);
    return hack.get();
}

Strangely, Eclipse has the same problem, but it gives more helpful output: For the above example, when str is @Nullable it infers the Function to be <? super @Nullable T, @Nullable U> but if str is not annotated it infers the Function to be <? super T, @Nullable U>.

Are IntelliJ's and Eclipse's analysers related in any way? What's used exactly in IntelliJ, is it something proprietary developed by JetBrains or is it maybe related to the Fernflower decompiler?

Again, thanks for taking the time to look into this, Peter!

Yes, it definitely looks like an issue with type inference (T is inferred as @Nullable String). Since 2017.1, there have been some fixes in that area with regard to nullability, Could you please check if you can reproduce the issue with 2017.2 public preview (https://blog.jetbrains.com/idea/2017/06/intellij-idea-2017-2-public-preview/)?

It's interesting to know about Eclipse, thanks! The code is independent, but it implements the same language spec, and so it's not very surprising that it produces similar results. IDEA's type inference is developed by JetBrains, and not related to Fernflower. I wouldn't call it very much proprietary, since it's open source.

Alright.

Unfortunately I exceeded the time budget for this so I'll have to postpone trying the public preview until next month.

Thanks for your help!

I just tested it wit 2017.3, the Problem seems to be resolved! Thanks, cased closed :)