Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correct duplicate ping_times during EK60 conversion (fixes #235) #433

Merged
merged 15 commits into from
Sep 23, 2021

Conversation

imranmaj
Copy link
Contributor

Fixes #235

Corrects duplicate ping_times during EK60 conversion by incrementing duplicate times by 1ns

@imranmaj imranmaj requested a review from leewujung August 26, 2021 00:05
@codecov-commenter
Copy link

codecov-commenter commented Aug 26, 2021

Codecov Report

Merging #433 (0340fb2) into dev (0a81a47) will decrease coverage by 6.16%.
The diff coverage is 70.73%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev     #433      +/-   ##
==========================================
- Coverage   76.41%   70.25%   -6.17%     
==========================================
  Files          37       22      -15     
  Lines        3227     2585     -642     
==========================================
- Hits         2466     1816     -650     
- Misses        761      769       +8     
Flag Coverage Δ
unittests 70.25% <70.73%> (-6.17%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
echopype/echodata/combine.py 60.34% <0.00%> (-11.98%) ⬇️
echopype/convert/set_groups_ek60.py 91.08% <76.31%> (-8.92%) ⬇️
echopype/__init__.py 0.00% <0.00%> (-100.00%) ⬇️
echopype/echodata/__init__.py 0.00% <0.00%> (-100.00%) ⬇️
echopype/echodata/convention/__init__.py 0.00% <0.00%> (-100.00%) ⬇️
echopype/echodata/api.py 16.66% <0.00%> (-66.67%) ⬇️
echopype/convert/convert.py 60.00% <0.00%> (-40.00%) ⬇️
echopype/echodata/echodata.py 54.74% <0.00%> (-32.41%) ⬇️
echopype/convert/api.py 80.83% <0.00%> (-5.00%) ⬇️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a81a47...0340fb2. Read the comment docs.

@leewujung
Copy link
Member

From discussions with @imranmaj and @emiliom :

  • we will keep a single data variable under the Provenance group that contains the original ping time
  • we will add additional attributes to record what problems had occur: reversal, duplicate
  • we will create methods (in a separate PR) that spit out the indices for which each of the problem occur

@leewujung leewujung merged commit 33c96c3 into OSOceanAcoustics:dev Sep 23, 2021
@leewujung leewujung added this to the 0.5.4 release milestone Sep 27, 2021
leewujung added a commit that referenced this pull request Oct 1, 2021
* re-attach sonar_model to combined echodata object (#437)

* Correct duplicate ping_times during EK60 conversion (fixes #235) (#433)

* Add fix for duplicate ping_time

* Remove dead code

* Increment by 1ns instead of 1ms

* Add duplicate ping time warning and store in provenance

* Autoformatter

* Add test for duplicate ping times file

* Add qc method to remove duplicate pings

* Drop entire pings with duplicate values

* Store entire original ping_time and add attribute when there are duplicate ping_times

* Remove duplicate ping_time qc method, it will be added back in future PR

* Change EchoData combine ping_time reversal attribute to netCDF encodable value

* Remove unused import

* Update tests

* Clarify duplicate ping_time removal and drop warnings

* Add removal note to deprecation warnings (#443)

* Add removal note to deprecation warnings

* Say removal will be specifically in next release

* AD2CP conversion bug fixes (#438)

* Allow strings in AD2CP config string data record

* Add ping_time dimension to ahrs variables

* Default AHRS to empty list instead of None

* Change AHRS dims when it does not exist

* Add update_platform method to EchoData (#434)

* Add update_platform method to EchoData

* Add test for update_platform

* Use drop_vars instead of drop

* Use .data on variables

* Generalize extra_platform_data time dimension name

* Add AZFP support

* Generalized update_platform handling of CF trajectory-encoded datasets

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Default water_level to zeros instead of ones

* Add warning for overwritten variables

* Index incoming data by time2

* Add long_name for time2

* Show dropped variable warning when nans do not exist

* Reword dropped variable warning

* Rename mapping_get_multiple to mapping_search_variable

Co-authored-by: Emilio Mayorga <emiliomayorga@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add summary statistics (#444)

* initial add of echo metrics functions

* update echo metrics to use range as data variable

* small readability edit

* added test functions to echopype.tests

* Modified syntax for consistency

* lint corrections on metrics

* added docstrings

* changed summary_statistics.py docstrings to numpy format

* Clean up tabs in code

* Clean up tabs in test code

* add metrics to run-test.py

* renamed test_metrics_summary_statistics

* small debug edits

* pre commit black edits

* moved test_metrics_summary_statistics.py to /tests/metrics/

* docstring update, add typing, change inertia to dispersion

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: stuara <stuara@uw.edu>
Co-authored-by: Landung "Don" Setiawan <landung.setiawan@gmail.com>
Co-authored-by: stuara <83487347+stuara@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Prepare workflows for master -> main (#445)

* Add PathNotFoundError in _load_convert for open_converted (#447)

* add PathNotFoundError in _load_convert for open_converted

Co-authored-by: imranmaj <49664304+imranmaj@users.noreply.github.com>
Co-authored-by: Emilio Mayorga <emiliomayorga@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: stuara <stuara@uw.edu>
Co-authored-by: Landung "Don" Setiawan <landung.setiawan@gmail.com>
Co-authored-by: stuara <83487347+stuara@users.noreply.github.com>
Co-authored-by: Landung "Don" Setiawan <landungs@uw.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants