Passing data to an Action

I have my ToolWindowFactory: 

class MyToolWindowFactory implements com.intellij.openapi.wm.ToolWindowFactory

In which I have some data, could be a String or an Object

I want to be able to retrieve it at the time my action will execute:

public class RefreshAction extends AnAction {

  @Override
  public void actionPerformed(@NotNull AnActionEvent e) {
    // Get data here
    e.getData(MyKey);
  }
}

In my ToolWindowFactory I have tried to inject data into the `DataContext`:

DataContext dataContext = DataManager.getInstance().getDataContext(toolWindow.getComponent());
DataManager.getInstance().saveInDataContext(dataContext, MyKey, MyData);

However this is not working.
I have also tried to use DataContext in the action itself:

@Override
public void actionPerformed(@NotNull AnActionEvent e) {
    DataContext dataContext = DataManager.getInstance().getDataContext();
    Object hello = DataManager.getInstance().loadFromDataContext(dataContext, DataKeys.LATEST_ERRORS_STATE_SUBJECT);
}

This did not work either.

Finally I have tried to implement DataProvider:

public class MyToolWindowFactory implements com.intellij.openapi.wm.ToolWindowFactory, DataProvider

And override the method to return some dummy data:

@Override
public @Nullable Object getData(@NotNull String dataId) {
    System.out.println("called");
    System.out.println(dataId);
    return "hello";
}

This method doesn't even get called.

Now I am quite sure I am missing something and doing something wrong, however there is basically no documentation or examples on this topic, so I am really improvising things.

Is what I want to achieve even possible?

Thanks