Skip to content

fix(http-client-csharp): preserve explicit null for optional nullable properties - #11971

Open
JoshLove-msft wants to merge 16 commits into
microsoft:mainfrom
JoshLove-msft:joshlove-msft-optional-nullable-presence
Open

JoshLove-msft wants to merge 16 commits into
microsoft:mainfrom
JoshLove-msft:joshlove-msft-optional-nullable-presence

Conversation

@JoshLove-msft

Copy link
Copy Markdown
Contributor

Summary

Fixes #11970.

Generated model classes now distinguish an untouched optional nullable property from one explicitly assigned null. Omitted properties remain omitted, explicit null writes JSON null, and non-null values keep their existing representation. JSON deserialization preserves these states for subsequent JSON and wire-format serialization.

The generator uses private backing storage and nonpublic presence flags for scalar/model properties, and reuses change-tracking collection state for nullable lists and dictionaries. It preserves inherited properties, including dynamic models with a non-dynamic base, without changing public property types or constructor/factory signatures. Expected projects were regenerated with the supported scripts; generated files were not hand-edited.

Compatibility boundaries

  • Required-nullable, optional-nonnullable, read-only wire filtering, XML serialization, and JsonPatch override precedence retain their existing behavior.
  • Defaulted constructor/factory scalar arguments retain null-as-omitted semantics. Existing factory collection initialization also remains unchanged, including default lists materialized as empty lists. Assign a writable property after construction to request explicit null.
  • Handwritten replacement properties/fields and models customized as readonly structs retain their existing behavior; no public presence-tracking API or replacement-member opt-in is introduced. These boundaries are documented in the customization guide.

Coverage and validation

Regression coverage includes omitted/null/value states, assignment transitions, model references, strings, integers, booleans, enums, dates, bytes, nullable collection elements, nested wire serialization, both JSON interfaces, round trips, duplicate-property last-value-wins, inheritance, name collisions, handwritten customization, factory defaults, read-only properties, and dynamic patches.

  • 4,538 generator/local unit tests passed.
  • 31 real Spector nullable and JSON merge-patch tests passed; test libraries were regenerated back to their standard stub form afterward.
  • 244 emitter tests passed, with 2 existing skips (--maxWorkers=4 avoids contention against the existing five-second timeout).
  • Full Generate.ps1, focused final regeneration, npm run build, API extraction, repository/emitter formatting, and C# formatting passed.
  • Cop checks passed. Repository oxlint scanned 70 emitter files with zero diagnostics; the package's existing npm run lint still references removed ESLint tooling, so the repository's current linter was used instead.

No C# changelog/Chronus entry is included.

Track generated nullable property presence independently of its value and preserve absent/null collection states during JSON deserialization. Cover assignment, round trips, inheritance, customization, factory defaults, and dynamic patch precedence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pkg-pr-new

pkg-pr-new Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/http-client-csharp@11971

commit: 2523391

@github-actions

Copy link
Copy Markdown
Contributor

No changes needing a change description found.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds presence tracking so C# models preserve omitted, explicit null, and non-null states during JSON and wire serialization.

Changes:

  • Adds backing fields and presence flags for nullable scalar/model properties.
  • Preserves nullable collection and dynamic-model states.
  • Adds regenerated samples, regression tests, and customization documentation.
File summaries
File Description
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/SampleTypeSpecModelFactory.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/Thing.Serialization.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/Thing.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/SampleTypeSpecContext.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableProperties.Serialization.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableProperties.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableDynamicProperties.Serialization.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableDynamicProperties.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableContainer.Serialization.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableContainer.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableChild.Serialization.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableChild.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableBase.Serialization.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableBase.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/NullableDynamicModel.Serialization.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/NullableDynamicModel.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/DynamicModel.Serialization.cs Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/Sample-TypeSpec.tsp Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local.Tests/ModelSerializationExtensionsTests.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/ScmModelProvider/TestData/ScmModelProviderTests/OptionalNullableCustomPropertyRetainsHandwrittenBehavior/Model.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/ScmModelProvider/ScmModelProviderTests.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanReplaceSerializationMethod.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanReplaceDeserializationMethod.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanCustomizeSerializationMethod.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanCustomizeDeserializationMethodWithoutOptions.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanCustomizeDeserializationMethodWithOptions.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/MrwSerializationTypeDefinitionTests/SerializedNameIsUsed(False).cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/ScmModelProvider.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/MrwSerializationTypeDefinition.Xml.cs Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/MrwSerializationTypeDefinition.cs Updated as part of this pull request.
packages/http-client-csharp/.tspd/docs/customization.md Updated as part of this pull request.
Review details

Suppressed comments (2)

packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/ScmModelProvider.cs:209

  • This treats every redeclared inherited property as sharing the base presence flag, even when ModelProvider marks the derived member new for a domain change (ModelProvider.cs:704-719). The generated new property keeps its field-based setter because this update only rewrites Override setters, so assigning null or a value never sets the shared flag and serialization omits the property; share the flag only for overrides and let new properties allocate their own state.
            if (property.BaseProperty != null)
            {
                var baseProperty = BaseModelProvider?.Properties.FirstOrDefault(p => p.Name == property.BaseProperty.Name);
                if (baseProperty != null && GetNullablePropertyPresence(baseProperty) is { } basePresence)
                {
                    _nullablePropertyPresence[property] = basePresence;
                    if (property.Modifiers.HasFlag(MethodSignatureModifiers.Override))
                    {
                        property.Update(body: new MethodPropertyBody(
                            Return(Base.Property(baseProperty.Name)),
                            property.Body.HasSetter ? Base.Property(baseProperty.Name).Assign(Value).Terminate() : null));
                    }
                }
                return;

packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/ScmModelProvider.cs:173

  • This reservation only includes generated m.Fields, but inherited handwritten fields are kept in m.CustomCodeView.Fields and merged separately by CanonicalTypeProvider (Microsoft.TypeSpec.Generator/src/Providers/CanonicalTypeProvider.cs:246-247,305-306). A base customization with an accessible field such as _textIsDefined can therefore receive the same name as a generated presence field in a derived model, producing a field-hiding warning; this generator treats warnings as errors (generator/Directory.Build.props:29). Include inherited CustomCodeView.Fields in the reserved names and add an inheritance/custom-field regression test.
            return ancestors.SelectMany(m => m.Fields.Select(f => f.Name));
  • Files reviewed: 14/32 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 16, 2026 22:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Nullable collection properties lose explicit null presence and serialize as omitted.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 17/37 changed files
  • Comments generated: 1
  • Review effort level: Lite

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 30, 2026 02:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Boolean assertions need correction, and focused XML presence coverage is still missing.

Review effort: Lite
Findings: 1 High severity · 1 Low severity

Open (2)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 30, 2026 02:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The critical XML null-collection issue and moderate custom-base field collision remain unresolved.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (2)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 30, 2026 16:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Nullable additional properties currently lose explicit null values during deserialization and subsequent wire serialization.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Copilot AI lite review requested due to automatic review settings October 5, 2026 23:21
@JoshLove-msft

Copy link
Copy Markdown
Contributor Author

Self-review

Reviewed the full cumulative diff against main (ScmModelProvider.cs, MrwSerializationTypeDefinition.cs/.Xml.cs, the docs update, and all test changes).

Found and fixed: an unresolved reviewer thread on AssertJsonShape (ModelSerializationExtensionsTests.cs) that had been missed in prior feedback passes — the switch wasn't exhaustive over JsonValueKind. I verified empirically that JsonValueKind.True/False/Null are distinct enum values, so the existing top-level ValueKind equality check already caught mismatches (the specific "won't throw" claim didn't reproduce), but added explicit cases for those kinds plus a default branch asserting GetRawText() equality for defense-in-depth. Pushed in 17e8fdf, replied to and resolved the thread.

No other actionable findings:

  • No changeset needed — @typespec/http-client-csharp isn't in any .chronus version policy (confirmed in .chronus/config.yaml), consistent with this package's own no-changelog convention.
  • The XML null-collection guard fix (MrwSerializationTypeDefinition.Xml.cs, using IsConcreteCollection + explicit null check before the runtime IsCollectionDefined helper) is a correct, necessary consequence of collections now being able to hold an explicit null at runtime.
  • Test coverage is thorough and exercises the real generator entry points (CreateModel, generated-output snapshot comparisons) across inheritance, custom/handwritten members, readonly structs, additional-properties unions, and XML — including several edge cases (0/1/128 properties, dynamic models, collision naming).

Full ClientModel suite (2219), Local suite (546 incl. the new case), and npm run cop all pass.

--generated by Copilot

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Three moderate unresolved findings affect concrete collection omission semantics and inherited presence tracking.

Review effort: Lite
Findings: None

Resolved since last review (1)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Avoid initializing concrete optional collections to empty values

packages/​http-client-csharp/​generator/​Microsoft.TypeSpec.Generator.ClientModel/​src/​Providers/​MrwSerializationTypeDefinition.cs:1066

This initializes every optional-nullable collection to its PropertyInitializationType, but concrete collections cannot represent an undefined state: OptionalSnippets.IsCollectionDefined explicitly treats null as undefined for them. For a concrete List<T>/dictionary property, an absent JSON property now becomes an empty collection and the writer emits []/{} instead of omitting it. Restrict this initialization to non-concrete, non-ReadOnlyMemory collections that use change-tracking storage (or otherwise leave concrete collections as default).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings October 5, 2026 23:53
@JoshLove-msft

Copy link
Copy Markdown
Contributor Author

Addressed the previously missed concrete optional-collection finding in ecb13c5.

Optional-nullable collections are now eagerly initialized only when their generated type uses change-tracking/interface storage. Customized concrete collections (for example, List<T>) remain default when the JSON property is absent, so subsequent serialization omits them instead of emitting an empty collection.

Added a regression covering concrete list and dictionary customizations alongside IList<T>/IDictionary<TKey, TValue> change-tracking controls. The test failed before the fix (List<T> initialized to ChangeTrackingList<T>) and passes afterward.

Validation: package build/API extraction, full ClientModel suite (2,220), full Local suite (546), emitter tests (381 passed, 2 skipped), full regeneration, Cop, and formatting all pass.

--generated by Copilot

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

An unresolved critical custom-property collision can produce duplicate generated members and compilation failures.

Review effort: Lite
Findings: 1 High severity

Open (1)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings October 6, 2026 15:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Duplicate additional-property keys can throw instead of applying last-value-wins behavior.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings October 6, 2026 16:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

A moderate duplicate-key last-value-wins issue remains unresolved for nullable additional-property unions.

Review effort: Lite
Findings: 1 Medium severity

Open (1)
Resolved since last review (1)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings October 6, 2026 18:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Typed additional-property unions with standalone null members can lose null values during wire deserialization and round-trip serialization.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings October 6, 2026 23:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved collection omission and inherited-property collision issues block approval.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings October 7, 2026 00:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The broad generator and serialization changes require final human review.

Review effort: Lite
Findings: None

Resolved since last review (1)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

emitter:client:csharp Issue for the C# client emitter: @typespec/http-client-csharp

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[http-client-csharp] Optional nullable properties cannot emit explicit JSON null

3 participants