Repository navigation
Conversation
Six defects, each reproduced against master before fixing: - UDTFMvAvg kept windowSum as a transform() local, reducing the sliding mean to (new - evicted) / w; promote it to an instance field - UDTFMinMax appended timestamps unconditionally but values only when finite, desynchronising both tables after a NaN row; skip such rows - UDTFSegment stored collected points in static fields, letting overlapping queries erase each other's state; use instance fields - UDAFPercentile kept the exact-count maps in static Maps shared by the singleton operations object; move them to instance state and pass the owning UDF into the operations methods (UDTFDistinct pattern) - UDTFHistogram overflowed end-start to +Inf with default boundaries so every value landed in bucket 0; use an overflow-safe, monotonic mapping that keeps the original formula for finite gaps - AbstractGapFillOperator used currentTime < endTime in the terminal fill guard while the endpoint is inclusive everywhere else, dropping the gap row at endTime when the last real row sat at endTime - step New tests: 10 in library-udf (shared UdfTestFakes lifecycle harness), 1 in datanode (GapFillWoGroupWoMoOperatorTest#testGapFillWoGroupWoMoOperator3). mvn -pl library-udf test: 29/29; GapFillWoGroupWoMoOperatorTest: 3/3. spotless:check clean on both modules. Signed-off-by: NUAAwqy <146524024+NUAAwqy@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes 19304 — six defects, each reproduced against master before fixing and covered by new unit tests.
library-udf
dprofile— query-scoped state (4 files)UDTFMvAvg: promotewindowSumto an instance field and reset it inbeforeStart; it was atransform()local, so the "sliding mean" was actually(new − evicted) / w.UDTFMinMax: skip non-finite rows entirely so the timestamps/values tables stay in lockstep (same approach asUDTFZScore).UDTFSegment: move the two collected lists fromstaticto instance fields — UDTF instances are per-query, the static lists let overlapping queries erase each other.UDAFPercentile: move the 4 exact-count maps fromstaticto instance state;PercentileOperations.initialize/insertExactnow take the owningUDAFPercentile(same pattern asUDTFDistinct), so the shared singleton no longer reassigns static state.UDTFHistogram— overflow-safe bucketingKeep the original formula for finite gaps (zero behaviour change, covered by per-bucket assertions); when
end - startoverflows to ±Inf (auto boundaries) fall back to a stable interval-centre based mapping that is monotonic, reaches both endpoint buckets, and degenerates to the old clamp whenstart == end.calc-commons— inclusive gap-fill endpointhasRemainingGapInPreviousGroup()comparedcurrentTime < endTimewhile the group-switch path andfillGapstreatendTimeas inclusive; the last gap point atendTimewas silently dropped when the final real row sat atendTime − step.Tests: 10 new library-udf tests via a shared lifecycle-fake harness (
UdfTestFakes), 1 new datanode gap-fill operator case (testGapFillWoGroupWoMoOperator3) —mvn -pl library-udf test29/29 green, spotless clean;GapFillWoGroupWoMoOperatorTest3/3 green. The MvAvg and gap-fill fixes were also regression-checked by reverting the source change and observing the new tests fail, then re-applying.