pyarrow analysis unexpected argument for pa.Table.from_pandas

analysis always giving some type of "unexpected argument" warning for pa.Table.from_pandas

code runs properly ... anybody else run into this? I hate when analysis not green
schema = pa.schema(fields)
table = pa.Table.from_pandas(df=df, schema=schema)


pycharm professional 2018.3
pyarrow pyarrow==0.11.1 and 0.12.0
9 comments
Comment actions Permalink

Hi,

Can you give a minimal code snippet necessary to reproduce the issue in a new project (including imports) ?

0
Comment actions Permalink
import pandas as pd
import pyarrow as pa

df = pd.DataFrame()

fields = [
pa.field('id', pa.int64()),
]

schema = pa.schema(fields)

table = pa.Table.from_pandas(df=df, schema=schema)
0
Comment actions Permalink

I also get "unexpected argument" error, but the code doesn't run in my case, producing "KeyError: 'id'"

Changing

table = pa.Table.from_pandas(df=df, schema=schema)

to

table = pa.Table.from_pandas(df=df)

Eliminates the error. So the issue is probably in pandas/pyarrow package. I'm using pandas 0.24.0 and pyarrow 0.12.0

Python 3.6.7

 

To ensure there's no confusion with run configurations, please try running the snippet you provided from outside of PyCharm using the same interpreter and environment. In my case this produces KeyError: 'id' error I've mentioned.

0
Comment actions Permalink

sorry try this code ....

import pandas as pd
import pyarrow as pa

data = [1,2,3,4,5]
df = pd.DataFrame(data)
print(df)

fields = [
pa.field('id', pa.int64()),
]

schema = pa.schema(fields)

table = pa.Table.from_pandas(df=df, schema=schema)
print(table)

0
Comment actions Permalink

The updated snippet results in the same error, KeyError: 'id'

from_pandas method doesn't seem to like the second argument. Also, if you run code inspection on this code, it will also mention "parameter 'type_cls' unfilled", which seems to be mandatory according to pyarrow doc: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html

In the end, this code doesn't run and the issue is still in the code, not IDE.

I would again suggest to run it directly with interpreter outside of PyCharm to ensure you're not running incorrect configuration.

0
Comment actions Permalink

the program runs fine for me in pycharm or from the command line

$ cat dog
import pandas as pd
import pyarrow as pa

data = [1,2,3,4,5]
df = pd.DataFrame(data)
print(df)

fields = [
pa.field('id', pa.int64()),
]

schema = pa.schema(fields)

table = pa.Table.from_pandas(df=df, schema=schema)
print(table)

$ python3 dog
0
0 1
1 2
2 3
3 4
4 5
pyarrow.Table
0: int64
__index_level_0__: int64
metadata
--------
{b'pandas': b'{"index_columns": ["__index_level_0__"], "column_indexes": [{"na'
b'me": null, "field_name": null, "pandas_type": "int64", "numpy_ty'
b'pe": "int64", "metadata": null}], "columns": [{"name": "0", "fie'
b'ld_name": "0", "pandas_type": "int64", "numpy_type": "int64", "m'
b'etadata": null}, {"name": null, "field_name": "__index_level_0__'
b'", "pandas_type": "int64", "numpy_type": "int64", "metadata": nu'
b'll}], "pandas_version": "0.23.4"}'}

0
Comment actions Permalink

That's strange, but ok.

Can you please post the output of `pip list` from the same environment you run this code?

0
Comment actions Permalink

$ pip list
Package Version
----------------- -----------
asn1crypto 0.24.0
bleach 3.1.0
boto3 1.9.79
botocore 1.12.86
certifi 2018.11.29
cffi 1.11.5
chardet 3.0.4
cryptography 2.5
dintetl 0.0.3
docopt 0.6.2
docutils 0.14
idna 2.8
jmespath 0.9.3
numpy 1.16.0
pandas 0.23.4
pip 19.0.1
pkginfo 1.5.0.1
psycopg2-binary 2.7.6.1
py4j 0.10.7
pyarrow 0.11.1
pycparser 2.19
Pygments 2.3.1
pyOpenSSL 19.0.0
pypandoc 1.4
pyspark 2.4.0
pyspark-stubs 2.4.0.post1
python-dateutil 2.7.5
pytz 2018.9
readme-renderer 24.0
requests 2.21.0
requests-toolbelt 0.8.0
s3transfer 0.1.13
setuptools 40.6.3
six 1.12.0
tqdm 4.30.0
twine 1.12.1
urllib3 1.24.1
webencodings 0.5.1
wheel 0.30.0

0
Comment actions Permalink

Ok, thanks for the info. I've reproduced the issue and created a ticket in YouTrack: https://youtrack.jetbrains.com/issue/PY-33799

Please follow on it, let's see what the developers would say.

0

Please sign in to leave a comment.