Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPU memory monitor fails to log allocation. #5011

Closed
trivialfis opened this issue Nov 5, 2019 · 1 comment · Fixed by #6414
Closed

GPU memory monitor fails to log allocation. #5011

trivialfis opened this issue Nov 5, 2019 · 1 comment · Fixed by #6414

Comments

@trivialfis
Copy link
Member

trivialfis commented Nov 5, 2019

When setting verbosity to 3 the memory logger should log all allocation and deallocation on GPU. But I encountered the deallocatation error when input is cudf, saying that the pointer in SpasePage::data is never allocated, which is obviously not true.

This is a reminder for myself. Also @RAMitchell might have some inputs here.

def test_cudf():
    X = np.random.randn(kRows, kCols)
    y = np.random.randn(kRows)

    df = pd.DataFrame(X)

    df = cudf.from_pandas(df)

    dtrain = xgb.DMatrix(df, label=y)
    bst = xgb.train({'tree_method': 'gpu_hist', 'verbosity': 2},
                    dtrain=dtrain,
                    evals=[(dtrain, 'X')],
                    num_boost_round=2)

    print('Pickle')
    import pickle
    with open('model.bst', 'bw') as fd:
        pickle.dump(bst, fd)

    with open('model.bst', 'br') as fd:
        bst: xgb.Booster = pickle.load(fd)
        dtest = xgb.DMatrix(df, label=y)
        bst.set_param({'gpu_id': '0'})
        bst.predict(dtest)
@trivialfis
Copy link
Member Author

Found the cause ... verbosity is not effective during DMatrix construction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant