[Backport 2.x][Feature] Support SQL direct query in Observability (#988…

…) (#1072) * [Feature] Support SQL direct query in Observability (#988) * remove unused files Signed-off-by: Eric Wei <menwe@amazon.com> * missing snapshots Signed-off-by: Eric Wei <menwe@amazon.com> * remove unused files Signed-off-by: Eric Wei <menwe@amazon.com> * missing snapshots Signed-off-by: Eric Wei <menwe@amazon.com> * create generic use polling hook Signed-off-by: Eric <menwe@amazon.com> * add unit tests Signed-off-by: Eric <menwe@amazon.com> * remove logging Signed-off-by: Eric <menwe@amazon.com> * datasource registeration for observability Signed-off-by: Eric <menwe@amazon.com> * Manage datasources (#967) * fix name change bug and modify test to test behavior Signed-off-by: Derek Ho <dxho@amazon.com> * get rid of lint Signed-off-by: Derek Ho <dxho@amazon.com> * test for flyout Signed-off-by: Derek Ho <dxho@amazon.com> * flyout to medium size Signed-off-by: Derek Ho <dxho@amazon.com> * make accelerate extensible Signed-off-by: Derek Ho <dxho@amazon.com> * get datasources and hook up to pplservice Signed-off-by: Derek Ho <dxho@amazon.com> * get flint working Signed-off-by: Derek Ho <dxho@amazon.com> * add datasource page with steps and buttons on bottom bar Signed-off-by: Derek Ho <dxho@amazon.com> * datasources as a new plugin and mostly working Signed-off-by: Derek Ho <dxho@amazon.com> * hook up manage to show datasources call Signed-off-by: Derek Ho <dxho@amazon.com> * update two tables with descriptions Signed-off-by: Derek Ho <dxho@amazon.com> * make some updates to the page Signed-off-by: Derek Ho <dxho@amazon.com> * cleanup unused files for data connections Signed-off-by: Derek Ho <dxho@amazon.com> * cleanup and add overview panel columns Signed-off-by: Derek Ho <dxho@amazon.com> * render tabs Signed-off-by: Derek Ho <dxho@amazon.com> * add unit tests Signed-off-by: Derek Ho <dxho@amazon.com> * update data test subj and snapshot Signed-off-by: Derek Ho <dxho@amazon.com> * Add datasources to management overview Signed-off-by: Derek Ho <dxho@amazon.com> * remove spark logo and update snapshot Signed-off-by: Derek Ho <dxho@amazon.com> * refactor routes out Signed-off-by: Derek Ho <dxho@amazon.com> * separate out the roles Signed-off-by: Derek Ho <dxho@amazon.com> * bump version back to 3.0 Signed-off-by: Derek Ho <dxho@amazon.com> --------- Signed-off-by: Derek Ho <dxho@amazon.com> * Add acceleration management UI (#989) * add acceleration management UI skeleton Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * Create new documentation link for acc Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * fix typos and minor bugs Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update snapshot Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update window location to hash Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove unused headers Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * dummy data redirection for datasource selector and language selector Signed-off-by: Eric <menwe@amazon.com> * flint datasource registration Signed-off-by: Eric <menwe@amazon.com> * tempararily passing down setup deps for datasource Signed-off-by: Eric <menwe@amazon.com> * add node server endpoints for direct query and job status Signed-off-by: Eric <menwe@amazon.com> * add server side endpoints for direct query Signed-off-by: Eric <menwe@amazon.com> * add s3 datasource class Signed-off-by: Eric <menwe@amazon.com> * s3 datasource registeration Signed-off-by: Eric <menwe@amazon.com> * changes to use new endpoints Signed-off-by: Eric <menwe@amazon.com> * remove unused files Signed-off-by: Eric Wei <menwe@amazon.com> * missing snapshots Signed-off-by: Eric Wei <menwe@amazon.com> * initial direct query support in explorer Signed-off-by: Eric <menwe@amazon.com> * add datasource pluggable Signed-off-by: Eric <menwe@amazon.com> * remove unused files Signed-off-by: Eric Wei <menwe@amazon.com> * missing snapshots Signed-off-by: Eric Wei <menwe@amazon.com> * remove unused dependency as well as passing down pluggable Signed-off-by: Eric <menwe@amazon.com> * changes for adopting new endpoints Signed-off-by: Eric <menwe@amazon.com> * add initial commits for context swithing Signed-off-by: Eric <menwe@amazon.com> * add query running page Signed-off-by: Eric <menwe@amazon.com> * relayout as an effort of match look and feel Signed-off-by: Eric <menwe@amazon.com> * add changes for index pattern to work with context switch Signed-off-by: Eric <menwe@amazon.com> * remove search section from explorer home Signed-off-by: Eric <menwe@amazon.com> * visualization page for direct query Signed-off-by: Eric <menwe@amazon.com> * add direct query visualization page Signed-off-by: Eric <menwe@amazon.com> * merge main Signed-off-by: Eric <menwe@amazon.com> * add discover redirection Signed-off-by: Eric <menwe@amazon.com> * merge main and resolve conflicts Signed-off-by: Eric <menwe@amazon.com> * s3 datasource and layout changes Signed-off-by: Eric <menwe@amazon.com> * use newly changed handler name Signed-off-by: Eric <menwe@amazon.com> * fixes for sidebar and datasource Signed-off-by: Eric <menwe@amazon.com> * add fix for redirection issue Signed-off-by: Eric <menwe@amazon.com> * console/comments cleanup Signed-off-by: Eric <menwe@amazon.com> * remove outdated polling tests Signed-off-by: Eric <menwe@amazon.com> * delete few outdated tests to resolve testing issues and skipped search tests for now Signed-off-by: Eric <menwe@amazon.com> * adopt empty prompt Signed-off-by: Eric <menwe@amazon.com> * remove unused oui/eui component Signed-off-by: Eric <menwe@amazon.com> * adopt new endpoint cchanges Signed-off-by: Eric <menwe@amazon.com> * fix data grid issue Signed-off-by: Eric <menwe@amazon.com> * disable autosuggest for sql Signed-off-by: Eric <menwe@amazon.com> * remove manual link for SQL Signed-off-by: Eric <menwe@amazon.com> * query cancelling Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric Wei <menwe@amazon.com> Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Derek Ho <dxho@amazon.com> Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Co-authored-by: Derek Ho <dxho@amazon.com> Co-authored-by: Shenoy Pratik <sgguruda@amazon.com> (cherry picked from commit e21dcf2) * remove datasource routers Signed-off-by: Joshua Li <joshuali925@gmail.com> --------- Signed-off-by: Joshua Li <joshuali925@gmail.com> Co-authored-by: Eric Wei <menwe@amazon.com>
opensearch-project · Oct 4, 2023 · cefd57f · cefd57f
1 parent 620cdef
commit cefd57f
Show file tree

Hide file tree

Showing 38 changed files with 1,919 additions and 423 deletions.
diff --git a/common/constants/data_connections.ts b/common/constants/data_connections.ts
@@ -0,0 +1,20 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+import { DatasourceType } from '../../common/types/data_connections';
+
+export const OPENSEARCH_DOCUMENTATION_URL = 'https://opensearch.org/docs/latest/data-sources/index';
+
+export const OPENSEARCH_ACC_DOCUMENTATION_URL =
+  'https://opensearch.org/docs/latest/data-acceleration/index';
+export const QUERY_RESTRICTED = 'query-restricted';
+export const QUERY_ALL = 'query-all';
+
+export const DatasourceTypeToDisplayName: { [key in DatasourceType]: string } = {
+  PROMETHEUS: 'Prometheus',
+  S3GLUE: 'S3',
+};
+
+export type AuthMethod = 'noauth' | 'basicauth' | 'awssigv4';
diff --git a/common/constants/explorer.ts b/common/constants/explorer.ts
@@ -88,6 +88,7 @@ export const REDUX_EXPL_SLICE_QUERY_TABS = 'queryTabs';
 export const REDUX_EXPL_SLICE_VISUALIZATION = 'explorerVisualization';
 export const REDUX_EXPL_SLICE_COUNT_DISTRIBUTION = 'countDistributionVisualization';
 export const REDUX_EXPL_SLICE_PATTERNS = 'patterns';
+export const REDUX_EXPL_SLICE_SEARCH_META_DATA = 'searchMetaData';
 export const PLOTLY_GAUGE_COLUMN_NUMBER = 4;
 export const APP_ANALYTICS_TAB_ID_REGEX = /application-analytics-tab.+/;
 export const DEFAULT_AVAILABILITY_QUERY = 'stats count() by span( timestamp, 1h )';

diff --git a/common/constants/shared.ts b/common/constants/shared.ts
@@ -14,6 +14,8 @@ export const DSL_CAT = '/cat.indices';
 export const DSL_MAPPING = '/indices.getFieldMapping';
 export const OBSERVABILITY_BASE = '/api/observability';
 export const INTEGRATIONS_BASE = '/api/integrations';
+export const JOBS_BASE = '/query/jobs';
+export const DATACONNECTIONS_BASE = '/api/dataconnections';
 export const EVENT_ANALYTICS = '/event_analytics';
 export const SAVED_OBJECTS = '/saved_objects';
 export const SAVED_QUERY = '/query';
@@ -23,6 +25,9 @@ export const SAVED_VISUALIZATION = '/vis';
 export const PPL_ENDPOINT = '/_plugins/_ppl';
 export const SQL_ENDPOINT = '/_plugins/_sql';
 export const DSL_ENDPOINT = '/_plugins/_dsl';
+export const DATACONNECTIONS_ENDPOINT = '/_plugins/_query/_datasources';
+export const JOBS_ENDPOINT_BASE = '/_plugins/_async_query';
+export const JOB_RESULT_ENDPOINT = '/result';
 
 export const observabilityID = 'observability-logs';
 export const observabilityTitle = 'Observability';
@@ -227,3 +232,5 @@ export const VISUALIZATION_ERROR = {
   NO_DATA: 'No data found.',
   INVALID_DATA: 'Invalid visualization data',
 };
+
+export const S3_DATASOURCE_TYPE = 'S3_DATASOURCE';
diff --git a/public/components/app.tsx b/public/components/app.tsx
@@ -55,6 +55,7 @@ export const App = ({
   timestampUtils,
   queryManager,
   startPage,
+  dataSourcePluggables,
 }: ObservabilityAppDeps) => {
   const { chrome, http, notifications, savedObjects: coreSavedObjects } = CoreStartProp;
   const parentBreadcrumb = {
@@ -85,6 +86,7 @@ export const App = ({
             parentBreadcrumb={parentBreadcrumb}
             parentBreadcrumbs={[parentBreadcrumb]}
             setBreadcrumbs={chrome.setBreadcrumbs}
+            dataSourcePluggables={dataSourcePluggables}
           />
         </MetricsListener>
       </I18nProvider>

diff --git a/public/components/common/search/autocomplete.tsx b/public/components/common/search/autocomplete.tsx
@@ -28,6 +28,7 @@ interface AutocompleteProps extends IQueryBarProps {
   placeholder?: string;
   possibleCommands?: Array<{ label: string }>;
   append?: any;
+  isSuggestionDisabled?: boolean;
 }
 
 export const Autocomplete = (props: AutocompleteProps) => {
@@ -45,6 +46,7 @@ export const Autocomplete = (props: AutocompleteProps) => {
     placeholder = 'Enter PPL query',
     possibleCommands,
     append,
+    isSuggestionDisabled = false,
   } = props;
 
   const [autocompleteState, setAutocompleteState] = useState<AutocompleteState<AutocompleteItem>>({
@@ -143,7 +145,7 @@ export const Autocomplete = (props: AutocompleteProps) => {
         {...(panelsFilter && { append, fullWidth: true })}
         disabled={isDisabled}
       />
-      {autocompleteState.isOpen && (
+      {autocompleteState.isOpen && !isSuggestionDisabled && (
         <div
           className={[
             'aa-Panel',

diff --git a/public/components/common/search/search.test.tsx b/public/components/common/search/search.test.tsx
@@ -7,7 +7,7 @@ import { fireEvent, render } from '@testing-library/react';
 import React from 'react';
 import { Search } from './search';
 
-describe('Search bar', () => {
+describe.skip('Search bar', () => {
   it('handles query change', () => {
     const query = 'rawQuery';
     const tempQuery = 'rawQuery';
@@ -62,7 +62,7 @@ describe('Search bar', () => {
         popoverItems={popoverItems}
         isLiveTailOn={isLiveTailOn}
         countDistribution={countDistribution}
-        curVisId={'line'} 
+        curVisId={'line'}
         spanValue={false}
         setSubType={'metric'}
         setMetricMeasure={'hours (h)'}

diff --git a/public/components/common/search/search.tsx b/public/components/common/search/search.tsx
@@ -5,8 +5,9 @@
 
 import './search.scss';
 
-import React, { useState } from 'react';
-import { isEqual } from 'lodash';
+import React, { useState, useEffect } from 'react';
+import { useDispatch, useSelector } from 'react-redux';
+import { isEqual, lowerCase } from 'lodash';
 import {
   EuiFlexGroup,
   EuiButton,
@@ -18,16 +19,26 @@ import {
   EuiContextMenuPanel,
   EuiToolTip,
   EuiCallOut,
+  EuiComboBox,
 } from '@elastic/eui';
 import { DatePicker } from './date_picker';
 import '@algolia/autocomplete-theme-classic';
 import { Autocomplete } from './autocomplete';
 import { SavePanel } from '../../event_analytics/explorer/save_panel';
 import { PPLReferenceFlyout } from '../helpers';
 import { uiSettingsService } from '../../../../common/utils';
-import { APP_ANALYTICS_TAB_ID_REGEX } from '../../../../common/constants/explorer';
+import { APP_ANALYTICS_TAB_ID_REGEX, RAW_QUERY } from '../../../../common/constants/explorer';
 import { LiveTailButton, StopLiveButton } from '../live_tail/live_tail_button';
 import { PPL_SPAN_REGEX } from '../../../../common/constants/shared';
+import { coreRefs } from '../../../framework/core_refs';
+import { useFetchEvents } from '../../../components/event_analytics/hooks';
+import { SQLService } from '../../../services/requests/sql';
+import {
+  selectSearchMetaData,
+  update as updateSearchMetaData,
+} from '../../event_analytics/redux/slices/search_meta_data_slice';
+import { usePolling } from '../../../components/hooks/use_polling';
+import { changeQuery } from '../../../components/event_analytics/redux/slices/query_slice';
 export interface IQueryBarProps {
   query: string;
   tempQuery: string;
@@ -52,7 +63,6 @@ export const Search = (props: any) => {
     query,
     tempQuery,
     handleQueryChange,
-    handleQuerySearch,
     handleTimePickerChange,
     dslService,
     startTime,
@@ -85,11 +95,34 @@ export const Search = (props: any) => {
     liveTailName,
     curVisId,
     setSubType,
+    setIsQueryRunning,
   } = props;
 
+  const explorerSearchMetadata = useSelector(selectSearchMetaData)[tabId];
+  const dispatch = useDispatch();
   const appLogEvents = tabId.match(APP_ANALYTICS_TAB_ID_REGEX);
   const [isSavePanelOpen, setIsSavePanelOpen] = useState(false);
   const [isFlyoutVisible, setIsFlyoutVisible] = useState(false);
+  const [queryLang, setQueryLang] = useState([]);
+  const [jobId, setJobId] = useState('');
+  const sqlService = new SQLService(coreRefs.http);
+  const { application } = coreRefs;
+
+  const {
+    data: pollingResult,
+    loading: pollingLoading,
+    error: pollingError,
+    startPolling,
+    stopPolling,
+  } = usePolling<any, any>((params) => {
+    return sqlService.fetchWithJobId(params);
+  }, 5000);
+
+  const requestParams = { tabId };
+  const { getLiveTail, getEvents, getAvailableFields, dispatchOnGettingHis } = useFetchEvents({
+    pplService: new SQLService(coreRefs.http),
+    requestParams,
+  });
 
   const closeFlyout = () => {
     setIsFlyoutVisible(false);
@@ -129,6 +162,50 @@ export const Search = (props: any) => {
     />
   );
 
+  const handleQueryLanguageChange = (lang) => {
+    if (lang[0].label === 'DQL') {
+      return application.navigateToUrl(
+        `../app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'${explorerSearchMetadata.datasources[0].value}',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_q=(filters:!(),query:(language:kuery,query:''))`
+      );
+    }
+    dispatch(
+      updateSearchMetaData({
+        tabId,
+        data: { lang: lang[0].label },
+      })
+    );
+    setQueryLang(lang);
+  };
+
+  const onQuerySearch = (lang) => {
+    handleTimeRangePickerRefresh();
+  };
+
+  useEffect(() => {
+    if (pollingResult && (pollingResult.status === 'SUCCESS' || pollingResult.datarows)) {
+      // update page with data
+      dispatchOnGettingHis(pollingResult, '');
+      // stop polling
+      stopPolling();
+      setIsQueryRunning(false);
+    }
+  }, [pollingResult, pollingError]);
+
+  useEffect(() => {
+    if (explorerSearchMetadata.datasources?.[0]?.type === 'DEFAULT_INDEX_PATTERNS') {
+      const queryWithSelectedSource = `source = ${explorerSearchMetadata.datasources[0].label}`;
+      handleQueryChange(queryWithSelectedSource);
+      dispatch(
+        changeQuery({
+          tabId,
+          query: {
+            [RAW_QUERY]: queryWithSelectedSource,
+          },
+        })
+      );
+    }
+  }, [explorerSearchMetadata.datasources]);
+
   return (
     <div className="globalQueryBar">
       <EuiFlexGroup gutterSize="s" justifyContent="flexStart" alignItems="flexStart">
@@ -141,14 +218,25 @@ export const Search = (props: any) => {
             </EuiToolTip>
           </EuiFlexItem>
         )}
-        <EuiFlexItem key="search-bar" className="search-area">
+        <EuiFlexItem key="lang-selector" className="search-area" grow={1}>
+          <EuiComboBox
+            placeholder="No language selected yet"
+            options={[{ label: 'PPL' }, { label: 'DQL' }]}
+            selectedOptions={queryLang}
+            onChange={handleQueryLanguageChange}
+            singleSelection={{ asPlainText: true }}
+          />
+        </EuiFlexItem>
+        <EuiFlexItem key="search-bar" className="search-area" grow={5}>
           <Autocomplete
             key={'autocomplete-search-bar'}
             query={query}
             tempQuery={tempQuery}
             baseQuery={baseQuery}
             handleQueryChange={handleQueryChange}
-            handleQuerySearch={handleQuerySearch}
+            handleQuerySearch={() => {
+              onQuerySearch(queryLang);
+            }}
             dslService={dslService}
             getSuggestions={getSuggestions}
             onItemSelect={onItemSelect}
@@ -178,7 +266,9 @@ export const Search = (props: any) => {
               liveStreamChecked={props.liveStreamChecked}
               onLiveStreamChange={props.onLiveStreamChange}
               handleTimePickerChange={(timeRange: string[]) => handleTimePickerChange(timeRange)}
-              handleTimeRangePickerRefresh={handleTimeRangePickerRefresh}
+              handleTimeRangePickerRefresh={() => {
+                onQuerySearch(queryLang);
+              }}
             />
           )}
         </EuiFlexItem>