fix: use flexible datetime parsing for start_date in file-based connectors by aaronsteers · Pull Request #887 · airbytehq/airbyte-python-cdk

Aaron ("AJ") Steers (aaronsteers) · 2026-01-30T22:51:16Z

Summary

Replaces the strict regex pattern for start_date validation in AbstractFileBasedSpec with a Pydantic validator using ab_datetime_try_parse. This fixes issues where valid ISO8601/RFC3339 datetime strings like 2025-01-01T00:00:00Z (without microseconds) were incorrectly rejected.

Before: Only accepted YYYY-MM-DDTHH:mm:ss.SSSSSSZ (exactly 6 microsecond digits required)
After: Accepts any valid ISO8601/RFC3339 format via ab_datetime_try_parse

Fixes: airbytehq/oncall#9390

Updates since last revision

Updated csv_scenarios.py test expectations to remove pattern and pattern_descriptor fields from the expected spec, aligning with the new Field definition

Review & Testing Checklist for Human

Schema impact: The pattern and pattern_descriptor fields were removed from the Field definition. Verify this doesn't break UI rendering or connector spec generation for file-based sources.
Validation permissiveness: ab_datetime_try_parse is quite flexible (accepts date-only, timestamps, various offsets). Confirm this level of permissiveness is acceptable for start_date.
Test with affected connectors: Test a file-based connector (e.g., S3, SharePoint) with various start_date formats including 2025-01-01T00:00:00Z and 2025-01-01T00:00:00.000000Z.

Notes

No new unit tests were added for the validator itself - existing file-based config tests pass with updated expectations
This affects all connectors inheriting from AbstractFileBasedSpec (S3, Azure Blob, SharePoint, etc.)
Link to Devin run: https://app.devin.ai/sessions/0f8d3c101b64494c813d5e711eea2662
Requested by: Aaron ("AJ") Steers (@aaronsteers)

Summary by CodeRabbit

New Features
- Enhanced start_date field validation to accept flexible date/time formats, including date-only entries, various time precisions, and timezone offsets.
- Improved error guidance for invalid date inputs.
Tests
- Added comprehensive validation tests for start_date format handling across multiple scenarios.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

…ctors Replace strict regex pattern with ab_datetime_try_parse validator to accept any valid ISO8601/RFC3339 datetime format. This fixes issues where valid datetime strings like '2025-01-01T00:00:00Z' (without microseconds) were incorrectly rejected. Fixes: airbytehq/oncall#9390 Co-Authored-By: AJ Steers <aj@airbyte.io>

devin-ai-integration · 2026-01-30T22:51:22Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

github-actions · 2026-01-30T22:51:27Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1769813348-fix-start-date-pattern#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1769813348-fix-start-date-pattern

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

/autofix - Fixes most formatting and linting issues
/poetry-lock - Updates poetry.lock file
/test - Runs connector tests with the updated CDK
/prerelease - Triggers a prerelease publish with default arguments
/poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
/poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

CDK API Reference

📝 Edit this welcome message.

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py

github-actions · 2026-01-30T22:58:22Z

PyTest Results (Fast)

3 848 tests +9 3 836 ✅ +9 6m 33s ⏱️ -15s
1 suites ±0 12 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit a29abec. ± Comparison against base commit 313db66.

♻️ This comment has been updated with latest results.

Co-Authored-By: AJ Steers <aj@airbyte.io>

github-actions · 2026-01-30T23:12:31Z

PyTest Results (Full)

3 851 tests +9 3 839 ✅ +9 10m 52s ⏱️ -10s
1 suites ±0 12 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit a29abec. ± Comparison against base commit 313db66.

♻️ This comment has been updated with latest results.

Co-Authored-By: AJ Steers <aj@airbyte.io>

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py

unit_tests/sources/file_based/config/test_abstract_file_based_spec.py

Co-Authored-By: AJ Steers <aj@airbyte.io>

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py

Co-Authored-By: AJ Steers <aj@airbyte.io>

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py

Co-Authored-By: AJ Steers <aj@airbyte.io>

Copilot

Pull request overview

This PR replaces strict regex validation for start_date in file-based connectors with a flexible Pydantic validator using ab_datetime_try_parse. This allows valid ISO8601/RFC3339 datetime formats (like 2025-01-01T00:00:00Z) that were previously rejected due to the requirement of exactly 6 microsecond digits.

Changes:

Added a Pydantic validator to AbstractFileBasedSpec that uses ab_datetime_try_parse for flexible datetime parsing
Updated the pattern and pattern_descriptor fields to reflect the broader range of accepted formats
Added test coverage for various datetime format inputs including edge cases

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py	Added validator method and updated field metadata to support flexible datetime formats
unit_tests/sources/file_based/config/test_abstract_file_based_spec.py	Added comprehensive test coverage for the new validator with various datetime formats
unit_tests/sources/file_based/scenarios/csv_scenarios.py	Updated test expectations to match new pattern and examples

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

unit_tests/sources/file_based/config/test_abstract_file_based_spec.py

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py

coderabbitai · 2026-01-30T23:41:13Z

📝 Walkthrough

Walkthrough

This PR enhances start_date validation in file-based source configurations by introducing Pydantic validators to enforce ISO8601/RFC3339 format parsing, expanding supported date-time formats, and adding comprehensive test coverage.

Changes

Cohort / File(s)	Summary
Start Date Validation Logic `airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py`	Adds `validator` from pydantic.v1 and `ab_datetime_try_parse` helper. Introduces `validate_start_date()` classmethod that enforces ISO8601/RFC3339 format parsing at runtime, raising ValueError for unparseable strings. Expands start_date field constraints with additional examples and a more permissive regex pattern accepting optional time components and timezones.
Test Coverage & Configuration `unit_tests/sources/file_based/config/test_abstract_file_based_spec.py`, `unit_tests/sources/file_based/scenarios/csv_scenarios.py`	Adds parametrized test `test_start_date_validation` covering valid (microseconds, milliseconds, timezone offsets, date-only, None) and invalid format cases. Updates CSV scenario start_date schema with expanded examples and relaxed regex pattern to accept date-only and various time precision formats.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: replacing strict regex validation with flexible datetime parsing for start_date in file-based connectors.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch devin/1769813348-fix-start-date-pattern

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

devin-ai-integration

Devin Review found 1 potential issue.

View issue and 3 additional flags in Devin Review.

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py

Aaron ("AJ") Steers (aaronsteers) · 2026-01-30T23:41:54Z

/prerelease

Prerelease Job Info

This job triggers the publish workflow with default arguments to create a prerelease.

Prerelease job started... Check job output.

✅ Prerelease workflow triggered successfully.

View the publish workflow run: https://github.com/airbytehq/airbyte-python-cdk/actions/runs/21534542274

Aldo Gonzalez (aldogonzalez8)

APPROVED

Co-Authored-By: AJ Steers <aj@airbyte.io>

Christo Grabowski (ChristoGrab)

devin-ai-integration · 2026-01-30T23:48:56Z

Validation Evidence: Flexible `start_date` Parsing

I've validated that the flexible datetime parsing works correctly on the SharePoint Enterprise connector (which inherits from AbstractFileBasedSpec).

Unit Tests Added

Added 9 parametrized test cases in unit_tests/test_spec.py covering:

With microseconds: 2021-01-01T00:00:00.000000Z
Without microseconds: 2021-01-01T00:00:00Z (Terraform provider format)
With milliseconds: 2021-01-01T00:00:00.000Z
With timezone offset: 2021-01-01T00:00:00+00:00
Date only: 2021-01-01
None value
Invalid strings (correctly rejected)
Empty strings (correctly rejected)

Direct Validation Results

Testing start_date validation on SourceMicrosoftSharePointSpec:
======================================================================
PASS: with_microseconds: "2021-01-01T00:00:00.000000Z" -> accepted
PASS: without_microseconds (Terraform format): "2021-01-01T00:00:00Z" -> accepted
PASS: with_milliseconds: "2021-01-01T00:00:00.000Z" -> accepted
PASS: terraform_provider_format: "2025-01-01T00:00:00Z" -> accepted
PASS: with_timezone_offset: "2021-01-01T00:00:00+00:00" -> accepted
PASS: date_only: "2021-01-01" -> accepted
PASS: none_value: "None" -> accepted
PASS: invalid_string: "not-a-date" -> correctly rejected
PASS: empty_string: "" -> correctly rejected
======================================================================
All tests passed!

This confirms the fix resolves the original issue where Terraform provider's Go time.Time serialization (without microseconds) was being rejected.

devin-ai-integration bot assigned Aaron ("AJ") Steers (aaronsteers) Jan 30, 2026

github-code-quality bot found potential problems Jan 30, 2026

View reviewed changes

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py Fixed Show fixed Hide fixed

test: update expected spec to remove pattern field for start_date

6886ab2

Co-Authored-By: AJ Steers <aj@airbyte.io>

test: add unit tests for start_date validation

198de6d

Co-Authored-By: AJ Steers <aj@airbyte.io>

Aaron ("AJ") Steers (aaronsteers) commented Jan 30, 2026

View reviewed changes

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py Show resolved Hide resolved

github-code-quality bot found potential problems Jan 30, 2026

View reviewed changes

unit_tests/sources/file_based/config/test_abstract_file_based_spec.py Fixed Show fixed Hide fixed

devin-ai-integration bot and others added 2 commits January 30, 2026 23:34

style: make validator signature multiline with noqa comment

cd5b9f0

Co-Authored-By: AJ Steers <aj@airbyte.io>

fix: remove unused PydanticValidationError import

8526249

Co-Authored-By: AJ Steers <aj@airbyte.io>

github-code-quality bot found potential problems Jan 30, 2026

View reviewed changes

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py Fixed Show fixed Hide fixed

coderabbitai bot approved these changes Jan 30, 2026

View reviewed changes

feat: add flexible pattern and multiple examples for start_date field

ef9c1dd

Co-Authored-By: AJ Steers <aj@airbyte.io>

Aaron ("AJ") Steers (aaronsteers) commented Jan 30, 2026

View reviewed changes

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py Show resolved Hide resolved

Aaron ("AJ") Steers (aaronsteers) marked this pull request as ready for review January 30, 2026 23:38

Copilot AI review requested due to automatic review settings January 30, 2026 23:38

style: reorder examples from shortest to longest

c807991

Co-Authored-By: AJ Steers <aj@airbyte.io>

Copilot AI reviewed Jan 30, 2026

View reviewed changes

unit_tests/sources/file_based/config/test_abstract_file_based_spec.py Outdated Show resolved Hide resolved

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py Show resolved Hide resolved

github-code-quality bot found potential problems Jan 30, 2026

View reviewed changes

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py Dismissed Show dismissed Hide dismissed

devin-ai-integration bot reviewed Jan 30, 2026

View reviewed changes

airbyte_cdk/sources/file_based/config/abstract_file_based_spec.py Show resolved Hide resolved

Aldo Gonzalez (aldogonzalez8) approved these changes Jan 30, 2026

View reviewed changes

test: remove slash separator test case (non-standard format)

a29abec

Co-Authored-By: AJ Steers <aj@airbyte.io>

Christo Grabowski (ChristoGrab) self-requested a review January 30, 2026 23:42

Christo Grabowski (ChristoGrab) approved these changes Jan 30, 2026

View reviewed changes

Aaron ("AJ") Steers (aaronsteers) merged commit efad73e into main Jan 31, 2026
28 of 29 checks passed

Aaron ("AJ") Steers (aaronsteers) deleted the devin/1769813348-fix-start-date-pattern branch January 31, 2026 00:04

Aaron ("AJ") Steers (aaronsteers) mentioned this pull request Feb 25, 2026

fix: use flexible datetime parsing in filter_files_by_globs_and_start_date #921

Open

4 tasks

devin-ai-integration bot mentioned this pull request Feb 25, 2026

test(source-microsoft-sharepoint): validate CDK datetime parsing fix airbytehq/airbyte#74044

Draft

1 task

Conversation

Aaron ("AJ") Steers (aaronsteers) commented Jan 30, 2026 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Updates since last revision

Review & Testing Checklist for Human

Notes

Summary by CodeRabbit

Uh oh!

devin-ai-integration bot commented Jan 30, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions bot commented Jan 30, 2026

👋 Greetings, Airbyte Team Member!

Testing This CDK Version

PR Slash Commands

Helpful Resources

Uh oh!

Uh oh!

github-actions bot commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Fast)

Uh oh!

github-actions bot commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Full)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai bot commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Aaron ("AJ") Steers (aaronsteers) commented Jan 30, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aldo Gonzalez (aldogonzalez8) left a comment

Choose a reason for hiding this comment

Uh oh!

Christo Grabowski (ChristoGrab) left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot commented Jan 30, 2026

Validation Evidence: Flexible start_date Parsing

Unit Tests Added

Direct Validation Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Aaron ("AJ") Steers (aaronsteers) commented Jan 30, 2026 •

edited by coderabbitai bot

Loading

github-actions bot commented Jan 30, 2026 •

edited

Loading

github-actions bot commented Jan 30, 2026 •

edited

Loading

coderabbitai bot commented Jan 30, 2026 •

edited

Loading

Aaron ("AJ") Steers (aaronsteers) commented Jan 30, 2026 •

edited by github-actions bot

Loading

Validation Evidence: Flexible `start_date` Parsing