Make SQL Server scaffolding compatible with Azure Data Explorer

[barnuri]

Fixes #34833

• I've read the guidelines for contributing and seen the walkthrough
• I've posted a comment on an issue with a detailed description of how I am planning to contribute and got approval from a member of the team
• The code builds and tests pass locally (also verified by our automated build checks)
• Commit messages follow this format:

        Summary of the changes
        - Detail 1
        - Detail 2

        Fixes #bugnumber

• Tests for the changes have been added (for bug fixes / features)
• Code follows the same patterns and style as existing code in this repo

[ErikEJ]

What are the results of

SELECT SERVERPROPERTY('Edition')

and

SELECT SERVERPROPERTY('EditionId')

[barnuri]

SERVERPROPERTY('EditionId')

tried that too, its sql azure

[ErikEJ]

@barnuri This looks valuable, I will get this added to EF Core Power Tools, including the CLI edition

[barnuri]

@barnuri please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@dotnet-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@dotnet-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@dotnet-policy-service agree company="Microsoft"

Contributor License Agreement

@dotnet-policy-service agree

[ErikEJ]

What tool are you using for coding this? I had a typo in my guess at spelling 😄

[barnuri]

What tool are you using for coding this? I had a typo in my guess at spelling 😄

using this scaffold cmd

dotnet ef dbcontext scaffold "$env:MyConnection" `
    Microsoft.EntityFrameworkCore.SqlServer `
    --output-dir "Models" `
    --context MyDbContext `
    --context-dir DAL `
    --no-onconfiguring `
    --table myTable `
    --force

oh you mean which ide i used, i used github web ^^
i will try now to build it locally because i see weird error in build

[ErikEJ]

LGTM

[ErikEJ]

@barnuri

I implemented a fix for this in the latest daily build, would be grateful if you could try it out.

ErikEJ/EFCorePowerTools@77275a5

[barnuri]

@barnuri

I implemented a fix for this in the latest daily build, would be grateful if you could try it out.

ErikEJ/EFCorePowerTools@77275a5

how can i use it ?
there is a way to install this version to dotnet tool ? and use it as dotnet ef dbcontext scaffold

[ErikEJ]

@barnuri Something similar, yes. Have a look at the readme here: https://www.nuget.org/packages/ErikEJ.EFCorePowerTools.Cli/9.1.409-nightly

[barnuri]

@barnuri Something similar, yes. Have a look at the readme here: https://www.nuget.org/packages/ErikEJ.EFCorePowerTools.Cli/9.1.409-nightly

steps:

1. dotnet tool install --global ErikEJ.EFCorePowerTools.Cli --version 8.1.409-nightly
2. efcpt "$myconnectionstring" mssql

got this error now

Microsoft.Data.SqlClient.SqlException: Request is invalid and cannot be processed: Semantic error: OTR0001: Reference to missing column 't.object_id' [line:position=11:29]
RequestId: TDS;ffcf7bdb-9f2b-4789-b4b8-7384289d2773;11
Time: 2024-10-06T17:38:03.5688957Z
  at void Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, bool breakConnection, Action<Action> wrapCloseInAction)
  at void Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, bool breakConnection, Action<Action> wrapCloseInAction)
  at void Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, bool callerHasConnectionLock, bool asyncClose)
  at bool Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject    
     stateObj, out bool dataReady)
  at bool Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
  at _SqlMetaDataSet Microsoft.Data.SqlClient.SqlDataReader.get_MetaData()
  at void Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, string resetOptionsString, bool isInternal, bool forDescribeParameterEncryption,  
     bool shouldCacheForAlwaysEncrypted)
  at SqlDataReader Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, bool returnStream, bool isAsync, int timeout, out Task task,  
     bool asyncWrite, bool inRetry, SqlDataReader ds, bool describeParameterEncryptionRequest)
  at SqlDataReader Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, bool returnStream, TaskCompletionSource<object> completion, int  
     timeout, out Task task, out bool usedCache, bool asyncWrite, bool inRetry, string method)
  at SqlDataReader Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, bool returnStream, string method)
  at SqlDataReader Microsoft.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
  at DbDataReader Microsoft.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
  at void Microsoft.EntityFrameworkCore.SqlServer.Scaffolding.Internal.PatchedSqlServerDatabaseModelFactory.GetTables(DbConnection connection, DatabaseModel databaseModel, Func<string,       
     string, string> tableFilter, IReadOnlyDictionary<string, ValueTuple<string, string>> typeAliases, string databaseCollation) in PatchedSqlServerDatabaseModelFactory.cs:650
  at DatabaseModel Microsoft.EntityFrameworkCore.SqlServer.Scaffolding.Internal.PatchedSqlServerDatabaseModelFactory.Create(DbConnection connection, DatabaseModelFactoryOptions options) in   
     PatchedSqlServerDatabaseModelFactory.cs:154
  at DatabaseModel Microsoft.EntityFrameworkCore.SqlServer.Scaffolding.Internal.PatchedSqlServerDatabaseModelFactory.Create(string connectionString, DatabaseModelFactoryOptions options) in   
     PatchedSqlServerDatabaseModelFactory.cs:112
  at DatabaseModel RevEng.Core.TableListBuilder.GetDatabaseModel() in TableListBuilder.cs:132
  at List<TableModel> RevEng.Core.TableListBuilder.GetTableModels() in TableListBuilder.cs:43
  at void ErikEJ.EFCorePowerTools.Services.DisplayService.<>c__DisplayClass7_0`1.<Wait>b__0(StatusContext ctx) in DisplayService.cs:64
  at Task Spectre.Console.Status.<>c__DisplayClass14_0.<Start>b__0(StatusContext ctx) in Status.cs:44
  at void Spectre.Console.Status.<>c__DisplayClass16_0.<<StartAsync>b__0>d.MoveNext() in Status.cs:79
  at void Spectre.Console.Status.<>c__DisplayClass17_0`1.<<StartAsync>b__0>d.MoveNext() in Status.cs:120
  at void Spectre.Console.Progress.<>c__DisplayClass32_0`1.<<StartAsync>b__0>d.MoveNext() in Progress.cs:138
  at async Task<T> Spectre.Console.Internal.DefaultExclusivityMode.RunAsync<T>(Func<Task<T>> func) in DefaultExclusivityMode.cs:40
  at async Task<T> Spectre.Console.Progress.StartAsync<T>(Func<ProgressContext, Task<T>> action) in Progress.cs:121
  at async Task<T> Spectre.Console.Status.StartAsync<T>(string status, Func<StatusContext, Task<T>> func) in Status.cs:117
  at async Task Spectre.Console.Status.StartAsync(string status, Func<StatusContext, Task> action) in Status.cs:77
  at void Spectre.Console.Status.Start(string status, Action<StatusContext> action) in Status.cs:48
  at T ErikEJ.EFCorePowerTools.Services.DisplayService.Wait<T>(string message, Func<T> doFunc) in DisplayService.cs:59
  at List<TableModel> ErikEJ.EFCorePowerTools.HostedServices.ScaffoldHostedService.GetTablesAndViews() in ScaffoldHostedService.cs:187
  at async Task ErikEJ.EFCorePowerTools.HostedServices.ScaffoldHostedService.ExecuteAsync(CancellationToken stoppingToken) in ScaffoldHostedService.cs:43
  at async Task Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
  at async Task Microsoft.Extensions.Hosting.Internal.Host.ForeachService<T>(IEnumerable<T> services, CancellationToken token, bool concurrent, bool abortOnFirstException, List<Exception>    
     exceptions, Func<T, CancellationToken, Task> operation)
  at void Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>g__LogAndRethrow|15_3(ref <>c__DisplayClass15_0 )
  at async Task Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
  at async Task<IHost> Microsoft.Extensions.Hosting.HostingAbstractionsHostBuilderExtensions.StartAsync(IHostBuilder hostBuilder, CancellationToken cancellationToken)
  at void ErikEJ.EFCorePowerTools.Program.<>c.<<MainAsync>b__1_1>d.MoveNext() in Program.cs:55
  at async Task<int> ErikEJ.EFCorePowerTools.Program.MainAsync(string[] args) in Program.cs:65
  at async Task<int> ErikEJ.EFCorePowerTools.Program.Main(string[] args) in Program.cs:22

[ErikEJ]

@roji @barnuri Maybe we should keep these changes in Power Tools for miw and see how it goes?

[roji]

I'm not against doing tweaks in EF to accomodate Azure Data Studio, but we need a wider understanding here, i.e. how compatible is ADS really with SQL Server, how usable will it be with EF, etc. Just adding scaffolding capabilities - either to EF or to the Power Tools - doesn't make much sense if nothing is going to work after that's done.

[ErikEJ]

Agree, but a way to find out is to enable queries against it based on a Power Tools generated DbContext - as an experimental feature..

[ErikEJ]

And it is Azure Data Explorer ( ADX)

[roji]

Agree, but a way to find out is to enable queries against it based on a Power Tools generated DbContext - as an experimental feature..

I don't think any sort of scaffolding or Power Tools support is needed here, just in order to test what ADX supports and whether it makes sense to use EF with it... That's just something that someone needs to investigate. But of course if you want to add a feature to Power Tools sure :)

[barnuri]

Agree, but a way to find out is to enable queries against it based on a Power Tools generated DbContext - as an experimental feature..

I don't think any sort of scaffolding or Power Tools support is needed here, just in order to test what ADX supports and whether it makes sense to use EF with it... That's just something that someone needs to investigate. But of course if you want to add a feature to Power Tools sure :)

after adding those fixes, i got full scaffolding of my db, the reason is to use adx via sql because adx doesnt have c# linq support and doesnt planning to have in the near future, and i want to connect it to some automatic tools like hot chocolate (graphql) to build strong api

[roji]

@barnuri OK, but what are the actual SQL capabilities of ADX? The subquery above which you converted into a JOIN is a pretty bad sign of the SQL capabilities being very limited.

[barnuri]

@barnuri OK, but what are the actual SQL capabilities of ADX? The subquery above which you converted into a JOIN is a pretty bad sign of the SQL capabilities being very limited.

you are right, but most of the time you dont use sub queries anyway because its had bad performance, and its support the most common simple queries that we need and always we can also make it better it adx too
anyway i think its very small changes that can make it works without dmg any other sql server so why not ?

[roji]

Is it only subqueries? Is there documentation anywhere saying what's supported and what isn't?

[roji]

Also note that tests seem to be failing.

[barnuri]

Also note that tests seem to be failing.

yeah missed some copy paste from EFCorePowerTools

Is it only subqueries? Is there documentation anywhere saying what's supported and what isn't?

from my experience yes, i verified all the basic and looks like its works great (select, where, joins group by)

[ErikEJ]

@roji @cincuranet Docs on Kusto SQL limitations: https://learn.microsoft.com/en-us/azure/data-explorer/t-sql#coverage

[roji]

Good work here @barnuri! This indeed replaces a correlated EXISTS subquery with a non-correlated IN. I'd actually accept that as an improvement regardless of the SQL limitations of ADX - we did this improvement in EF's query generation in 8.0, both because the SQL more clearly matches the intent and because it can be (dramatically) faster on some databases (but not on SQL Server).

[barnuri]

thanks :)
and thanks for the collaboration 💪🏾

[roji]

@ErikEJ @barnuri after the last modifications, I'm OK with merging the PR as it is. Has the appropriate smoke testing been done, @ErikEJ are you OK with this as well?

[ErikEJ]

Let me do a little smoke testing on the Power Tools PR that resembles this

[ErikEJ]

@roji @barnuri I have done some smoke testing with the EF Core Power Tools daily build, and everything works as expected.

I like the minimal footprint this PR has.

🐑 🇮🇹

[roji]

Merged for 10, thanks for your contribution @barnuri!