Added configuration management using pydantic #986
Conversation
add Pydantic configuration
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
Please check the codacy errors: https://app.codacy.com/gh/mlcommons/GaNDLF/pull-requests/986/issues |
sarthakpati
changed the title
|
@szmazurek could you please take a first pass? |
yup, will do in like 1hr or tomorrow morning |
| save_output: bool = Field( | ||
| default=False, description="Save outputs during validation/testing." | ||
| ) | ||
| in_memory: bool = Field(default=False, description="Pin data to CPU memory.") |
There was a problem hiding this comment.
What does it mean to "pin data to cpu/gpu memory"? Also, is the 'in_memory' really enforcing a page-lock on memory storing given chunk of data or just keeps it all in RAM?
There was a problem hiding this comment.
I took these default parameters
GaNDLF/GANDLF/config_manager.py
Line 16 in 3bfe133
and tried to manipulate them using Pydantic.
There was a problem hiding this comment.
pin_memory is used here:
GaNDLF/GANDLF/data/__init__.py
Line 28 in 3bfe133
There was a problem hiding this comment.
Regarding the comment about OOM with pin_memory (not really related to the pr tho, just in general) - pin memory may go OOM when the program runs at nearly full RAM utilization available within the machine, as page-locking will prevent parts of memory from being swapped, but I would advocate for allowing the user to try that option too (I did it in the lightning port)
| data_postprocessing: Union[dict, set] = Field( | ||
| default={}, description="Default data postprocessing configuration." | ||
| ) | ||
| grid_aggregator_overlap: str = Field( |
There was a problem hiding this comment.
What kind of other options can we have here? I believe this cannot be an arbitraty string, therefore it should be an optional literal here with available strings?
There was a problem hiding this comment.
I believe it can be either of these: ["crop", "average", "hann"]
Ref: https://torchio.readthedocs.io/patches/patch_inference.html#grid-aggregator
| model_config = ConfigDict( | ||
| extra="allow" | ||
| ) # it allows extra fields in the model dict | ||
| dimension: Optional[int] = Field(description="Dimension.") |
There was a problem hiding this comment.
Is the dimension optional? Also, maybe it should accept only 2 or 3, as no other dimensionalities are supported. And perhaps the description can be made more expressive - like 'model input dimension (2D or 3D).'?
There was a problem hiding this comment.
If the user doesn't define the dimension, it is calculated in the validate_patch_size(patch_size, dimension) function in the validators file using the patch size.
Hmm, yes, maybe it should only accept 2 or 3. I will change it and update the description.
There was a problem hiding this comment.
@benmalef: where is the function validate_patch_size? I am unable to find it in either the master or your branch.
There is a way to calculate the dimension automatically via ITK, though. But my guess is that this probably should be rolled into the overall cohort characteristics and sanity checks (#956).
There was a problem hiding this comment.
@sarthakpati ,Here is the function https://github.com/benmalef/GaNDLF/blob/8837b7ccd25b747c7cbe4faaa77226a631febd85/GANDLF/Configuration/Parameters/validators.py#L132. in my branch.
I tried to manipulate this code using the Pydantic.
GaNDLF/GANDLF/config_manager.py
Line 136 in 3bfe133
There was a problem hiding this comment.
Understood. I think we still need to ensure that the check for dimension is for 2 and 3 in the validation step itself so that we can give a meaningful error to the user.
| ) # it allows extra fields in the model dict | ||
| dimension: Optional[int] = Field(description="Dimension.") | ||
| architecture: Union[ARCHITECTURE_OPTIONS, dict] = Field(description="Architecture.") | ||
| final_layer: str = Field(description="Final layer.") |
There was a problem hiding this comment.
Here we are also limited to certain amount of acceptable values - leveraging literal seems like good option.
There was a problem hiding this comment.
Yesss, you are right. I change it . :P
| ), | ||
| default=3, | ||
| ) # TODO: check it | ||
| type: Optional[str] = Field(description="Type of model.", default="torch") |
There was a problem hiding this comment.
Should it also be literal? Probably options are torch, openvino? @sarthakpati am I right?
| default=3, | ||
| ) # TODO: check it | ||
| type: Optional[str] = Field(description="Type of model.", default="torch") | ||
| data_type: str = Field(description="Data type.", default="FP32") |
There was a problem hiding this comment.
Is this true that we support such field in the config and it really influences anything in base gandlf? I tought that precision is chaning only when amp is enabled
There was a problem hiding this comment.
I found it in the config_manager file and tried to manipulate it using pydantic .
GaNDLF/GANDLF/config_manager.py
Line 594 in 3bfe133
There was a problem hiding this comment.
I think @szmazurek is right. This is only changing when amp gets enabled. However, this flag is probably used by an earlier version of OpenVINO. Perhaps you can comment those lines out @benmalef and see if that makes a difference?
| type: Optional[str] = Field(description="Type of model.", default="torch") | ||
| data_type: str = Field(description="Data type.", default="FP32") | ||
| save_at_every_epoch: bool = Field(default=False, description="Save at every epoch.") | ||
| amp: bool = Field(default=False, description="Amplifier.") |
There was a problem hiding this comment.
amp stands for automatic mixed precision, not amplifier
There was a problem hiding this comment.
Yess, you are right. Sorry for this, the auto-complete sometimes messes up. I changed it
| default=-5, | ||
| description="this controls the number of validation data folds to be used for model *selection* during training (not used for back-propagation)", | ||
| ) | ||
| proportional: Optional[bool] = Field(default=None) |
There was a problem hiding this comment.
what does this parameter do? also, if it's boolean, can't we set default as False?
There was a problem hiding this comment.
I don't what does it, but i found it here
GaNDLF/GANDLF/config_manager.py
Line 638 in 3bfe133
So, I manipulate it using the def validate_nested_training(self) with the decorator model_validator.
I donlt know what is the default value. We may set the default as False.
There was a problem hiding this comment.
Yes, let's set this to False.
| description="this will perform stratified k-fold cross-validation but only with offline data splitting", | ||
| ) | ||
| testing: int = Field( | ||
| default=-5, |
There was a problem hiding this comment.
Open question - are there any limits to values that can be set in this field? Like, what happens if I set testing to 10? If there are limits, maybe it's worth to include possible ranges when defining this field, what do you think ? @sarthakpati @benmalef
There was a problem hiding this comment.
Hmm I don't know. Again, I found this value in the config_manager
GaNDLF/GANDLF/config_manager.py
Line 642 in 3bfe133
There was a problem hiding this comment.
I believe the only range is the total dataset number. For example, if we have 100 subjects, we cannot create 101 testing/validation folds. But this is an unrealistic example. Perhaps we can set some constraints - 10 seems appropriate.
|
|
||
|
|
||
| class PatchSampler(BaseModel): | ||
| type: str = Field(default="uniform") |
There was a problem hiding this comment.
are there any other options available for type and padding_mode? if so, maybe we should use literal here?
There was a problem hiding this comment.
Yes, I think this should be a literal. We only support uniform and label [ref].
| @model_validator(mode="after") | ||
| def validate_version(self) -> Self: | ||
| if version_check(self.model_dump(), version_to_check=version("GANDLF")): | ||
| return self |
There was a problem hiding this comment.
should we rise an error here if the condition is not met?
There was a problem hiding this comment.
Yes, we will rise an assertion error in the version_check fun.
GaNDLF/GANDLF/utils/generic.py
Line 91 in 3bfe133
| ) | ||
| # min_lr: 0.00001, #TODO: this should be defined ?? | ||
| # max_lr: 1, #TODO: this should be defined ?? | ||
| step_size: float = Field(description="step_size", default=None) |
There was a problem hiding this comment.
I think we need to think on different classes that would allow for definition of params for separate schedulers - for example, if we use a scheduler which does reduce on plateau, then we need a field to define tracked metric. Not really sure how to implement that nicely tho. Other approach is define all possible fields that any scheduler can take and later provide validation logic which takes care of conditionality - i.e if reduce_on_plateau type chosen, then we require monitor field
There was a problem hiding this comment.
hmm, if we could define different classes for each scheduler it would be great. We should think about it.
| class UserDefinedParameters(DefaultParameters): | ||
| version: Version = Field( | ||
| default=Version(minimum=version("GANDLF"), maximum=version("GANDLF")), | ||
| description="Whether weighted loss is to be used or not.", |
There was a problem hiding this comment.
Descritpion is not valid I believe :P
There was a problem hiding this comment.
hahah.... I changed :P
| patch_size: Union[list[Union[int, float]], int, float] = Field( | ||
| description="Patch size." | ||
| ) | ||
| model: Model = Field(..., description="The model to use. ") |
There was a problem hiding this comment.
should be list of avaiable strings no?
There was a problem hiding this comment.
I took this list
GaNDLF/GANDLF/models/__init__.py
Line 41 in 3bfe133
and I define it as literal in the model architecture parameter.
| description="Scheduler.", default=Scheduler(type="triangle_modified") | ||
| ) | ||
| optimizer: Union[str, Optimizer] = Field( | ||
| description="Optimizer.", default=Optimizer(type="adam") |
There was a problem hiding this comment.
Question about naming - here I first assumed we are initializing real torch optimizer - perhaps the names of config classes should be suffixed with Config/params?
There was a problem hiding this comment.
For the variable names, right? Absolutely - it would make it clear for developers.
There was a problem hiding this comment.
what about Optimizer_config ?
There was a problem hiding this comment.
Yup, it should be something along the lines of ${dict_name}_config for everything, even if it is something that isn't used as a variable later (such as optimizer, model, ...). Makes things clear for devs.
| data_postprocessing_after_reverse_one_hot_encoding: dict = Field( | ||
| description="data_postprocessing_after_reverse_one_hot_encoding.", default={} | ||
| ) | ||
| differential_privacy: Any = Field(description="Differential privacy.", default=None) |
There was a problem hiding this comment.
Can it be just a boolean field?
There was a problem hiding this comment.
This can be either a boolean or a dict:
differential_privacy:
max_grad_norm: 0.015625
noise_multiplier: 128.0
physical_batch_size: 64
| Field(description="Data preprocessing."), | ||
| AfterValidator(validate_data_preprocessing), | ||
| ] = {} | ||
| # TODO: It should be defined with a better way (using a BaseModel class) |
There was a problem hiding this comment.
I agree with the comments, it would allow for a lot of clarity
| file.write("\n".join(markdown)) | ||
|
|
||
|
|
||
| def initialize_key( |
There was a problem hiding this comment.
do we need such utility? Meaning, if there are default parameters to be set, ideally they are defined via pydantic and automatically populated if user did not set them explicitly
There was a problem hiding this comment.
Yes, you are right.
The best way is not to use it, but some parameters have complex logic, and I need to figure out how to implement them using Pydantic.
So, I use it for some parameters as they are in the config_manager.
For example, for the data_augmentation parameter, I use it in the validate_data_augmentation(value, patch_size) function in the validators file.
setup.py
Outdated
| @@ -85,6 +87,7 @@ | |||
| "openslide-bin", | |||
| "openslide-python==1.4.1", | |||
| "lion-pytorch==0.2.2", | |||
| "pydantic", | |||
There was a problem hiding this comment.
I would fix version, maybe in the future releases there are going to be some breaking changes of pydantic (little chance but I am paranoid a little)
There was a problem hiding this comment.
Done. Updated the Pydantic in the 2.10.6 version.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
@check-spelling-bot Report🔴 Please reviewSee the 📂 files view, the 📜action log, or 📝 job summary for details.Unrecognized words (1)hann Some files were automatically ignored 🙈These sample patterns would exclude them: You should consider adding them to: File matching is via Perl regular expressions. To check these files, more of their words need to be in the dictionary than not. You can use To accept these unrecognized words as correct and update file exclusions, you could run the following commands... in a clone of the [email protected]:benmalef/GaNDLF.git repository curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/main/apply.pl' |
perl - 'https://github.com/mlcommons/GaNDLF/actions/runs/13841289596/attempts/1'
Available 📚 dictionaries could cover words not in the 📘 dictionary
Consider adding them (in with:
extra_dictionaries: |
cspell:java/src/java-terms.txtTo stop checking additional dictionaries, add (in check_extra_dictionaries: ''Warnings (1)See the 📂 files view, the 📜action log, or 📝 job summary for details.
See |
add Pydantic configuration
Fixes #ISSUE_NUMBER
Proposed Changes
Checklist
CONTRIBUTINGguide has been followed.typingis used to provide type hints, including and not limited to usingOptionalif a variable has a pre-defined value).pip installstep is needed for PR to be functional), please ensure it is reflected in all the files that control the CI, namely: python-test.yml, and all docker files [1,2,3].logginglibrary is being used and noprintstatements are left.