Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set max field size limit for export token transfers #399

Merged
merged 1 commit into from
Oct 14, 2022

Conversation

m0ssc0de
Copy link
Contributor

@m0ssc0de m0ssc0de commented Oct 2, 2022

By execute ethereumetl extract_token_transfers --logs logs.csv --output token_transfers.csv --max-workers 1, we got error as below.

Traceback (most recent call last):
  File "/usr/bin/ethereumetl", line 33, in <module>
    sys.exit(load_entry_point('ethereum-etl==2.0.6', 'console_scripts', 'ethereumetl')())
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/ethereumetl/cli/extract_token_transfers.py", line 57, in extract_token_transfers
    job.run()
  File "/usr/lib/python3.10/site-packages/blockchainetl/jobs/base_job.py", line 28, in run
    self._export()
  File "/usr/lib/python3.10/site-packages/ethereumetl/jobs/extract_token_transfers_job.py", line 50, in _export
    self.batch_work_executor.execute(self.logs_iterable, self._extract_transfers)
  File "/usr/lib/python3.10/site-packages/ethereumetl/executors/batch_work_executor.py", line 58, in execute
    for batch in dynamic_batch_iterator(work_iterable, lambda: self.batch_size):
  File "/usr/lib/python3.10/site-packages/ethereumetl/utils.py", line 113, in dynamic_batch_iterator
    for item in iterable:
  File "/usr/lib/python3.10/csv.py", line 111, in __next__
    row = next(self.reader)
_csv.Error: field larger than field limit (131072)

After call function set_max_field_size_limit(), error disappeared.

@medvedev1088 medvedev1088 merged commit 688ecdf into blockchain-etl:develop Oct 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants