html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456222,IC_kwDOCGYnMM4_-6Re,9599,simonw,2022-03-21T03:45:52Z,2022-03-21T03:45:52Z,OWNER,Needs tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456155,IC_kwDOCGYnMM4_-6Qb,9599,simonw,2022-03-21T03:45:37Z,2022-03-21T03:45:37Z,OWNER,"Prototype:
```diff
diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
index 8255b56..0a3693e 100644
--- a/sqlite_utils/cli.py
+++ b/sqlite_utils/cli.py
@@ -2583,7 +2583,11 @@ def _generate_convert_help():
""""""
).strip()
recipe_names = [
- n for n in dir(recipes) if not n.startswith(""_"") and n not in (""json"", ""parser"")
+ n
+ for n in dir(recipes)
+ if not n.startswith(""_"")
+ and n not in (""json"", ""parser"")
+ and callable(getattr(recipes, n))
]
for name in recipe_names:
fn = getattr(recipes, name)
diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py
index 6918661..569c30d 100644
--- a/sqlite_utils/recipes.py
+++ b/sqlite_utils/recipes.py
@@ -1,17 +1,38 @@
from dateutil import parser
import json
+IGNORE = object()
+SET_NULL = object()
-def parsedate(value, dayfirst=False, yearfirst=False):
+
+def parsedate(value, dayfirst=False, yearfirst=False, errors=None):
""Parse a date and convert it to ISO date format: yyyy-mm-dd""
- return (
- parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()
- )
+ try:
+ return (
+ parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst)
+ .date()
+ .isoformat()
+ )
+ except parser.ParserError:
+ if errors is IGNORE:
+ return value
+ elif errors is SET_NULL:
+ return None
+ else:
+ raise
-def parsedatetime(value, dayfirst=False, yearfirst=False):
+def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None):
""Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS""
- return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat()
+ try:
+ return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat()
+ except parser.ParserError:
+ if errors is IGNORE:
+ return value
+ elif errors is SET_NULL:
+ return None
+ else:
+ raise
def jsonsplit(value, delimiter="","", type=str):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073455905,IC_kwDOCGYnMM4_-6Mh,9599,simonw,2022-03-21T03:44:47Z,2022-03-21T03:45:00Z,OWNER,"This is quite nice:
```
% sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.IGNORE)""
[####################################] 100%
% sqlite-utils rows test-dates.db dates
[{""id"": 1, ""date"": ""2016-03-15""},
{""id"": 2, ""date"": ""2016-03-16""},
{""id"": 3, ""date"": ""2016-03-17""},
{""id"": 4, ""date"": ""2016-03-18""},
{""id"": 5, ""date"": ""2016-03-19""},
{""id"": 6, ""date"": ""2016-03-20""},
{""id"": 7, ""date"": ""2016-03-21""},
{""id"": 8, ""date"": ""2016-03-22""},
{""id"": 9, ""date"": ""2016-03-23""},
{""id"": 10, ""date"": ""//""},
{""id"": 11, ""date"": ""2016-03-25""},
{""id"": 12, ""date"": ""2016-03-26""},
{""id"": 13, ""date"": ""2016-03-27""},
{""id"": 14, ""date"": ""2016-03-28""},
{""id"": 15, ""date"": ""2016-03-29""},
{""id"": 16, ""date"": ""2016-03-30""},
{""id"": 17, ""date"": ""2016-03-31""},
{""id"": 18, ""date"": ""2016-04-01""}]
% sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.SET_NULL)""
[####################################] 100%
% sqlite-utils rows test-dates.db dates
[{""id"": 1, ""date"": ""2016-03-15""},
{""id"": 2, ""date"": ""2016-03-16""},
{""id"": 3, ""date"": ""2016-03-17""},
{""id"": 4, ""date"": ""2016-03-18""},
{""id"": 5, ""date"": ""2016-03-19""},
{""id"": 6, ""date"": ""2016-03-20""},
{""id"": 7, ""date"": ""2016-03-21""},
{""id"": 8, ""date"": ""2016-03-22""},
{""id"": 9, ""date"": ""2016-03-23""},
{""id"": 10, ""date"": null},
{""id"": 11, ""date"": ""2016-03-25""},
{""id"": 12, ""date"": ""2016-03-26""},
{""id"": 13, ""date"": ""2016-03-27""},
{""id"": 14, ""date"": ""2016-03-28""},
{""id"": 15, ""date"": ""2016-03-29""},
{""id"": 16, ""date"": ""2016-03-30""},
{""id"": 17, ""date"": ""2016-03-31""},
{""id"": 18, ""date"": ""2016-04-01""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453370,IC_kwDOCGYnMM4_-5k6,9599,simonw,2022-03-21T03:41:06Z,2022-03-21T03:41:06Z,OWNER,I'm going to try the `errors=r.IGNORE` option and see what that looks like once implemented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453230,IC_kwDOCGYnMM4_-5iu,9599,simonw,2022-03-21T03:40:37Z,2022-03-21T03:40:37Z,OWNER,"I think the options here should be:
- On error, raise an exception and revert the transaction (the current default)
- On error, leave the value as-is
- On error, set the value to `None`
These need to be indicated by parameters to the `r.parsedate()` function.
Some design options:
- `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API.
- `errors=""ignore""`, `errors=""set-null""` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism
- `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL`
- `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073451659,IC_kwDOCGYnMM4_-5KL,9599,simonw,2022-03-21T03:35:01Z,2022-03-21T03:35:01Z,OWNER,"I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction.
Here's the code that does that:
https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073450588,IC_kwDOCGYnMM4_-45c,9599,simonw,2022-03-21T03:32:58Z,2022-03-21T03:32:58Z,OWNER,"Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted:
sqlite-utils convert test-dates.db dates date 'value.replace(""-"", ""/"")'
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates,
https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073448904,IC_kwDOCGYnMM4_-4fI,9599,simonw,2022-03-21T03:28:12Z,2022-03-21T03:30:37Z,OWNER,"Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/
```
sqlite-utils create-database test-dates.db
sqlite-utils create-table test-dates.db dates id integer date text --pk id
sqlite-utils test-dates.db ""WITH RECURSIVE
cnt(x) AS (
SELECT 0
UNION ALL
SELECT x+1 FROM cnt
LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1)
)
insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;""
```
After running that:
```
% sqlite-utils rows test-dates.db dates
[{""id"": 1, ""date"": ""2016-03-15""},
{""id"": 2, ""date"": ""2016-03-16""},
{""id"": 3, ""date"": ""2016-03-17""},
{""id"": 4, ""date"": ""2016-03-18""},
{""id"": 5, ""date"": ""2016-03-19""},
{""id"": 6, ""date"": ""2016-03-20""},
{""id"": 7, ""date"": ""2016-03-21""},
{""id"": 8, ""date"": ""2016-03-22""},
{""id"": 9, ""date"": ""2016-03-23""},
{""id"": 10, ""date"": ""2016-03-24""},
{""id"": 11, ""date"": ""2016-03-25""},
{""id"": 12, ""date"": ""2016-03-26""},
{""id"": 13, ""date"": ""2016-03-27""},
{""id"": 14, ""date"": ""2016-03-28""},
{""id"": 15, ""date"": ""2016-03-29""},
{""id"": 16, ""date"": ""2016-03-30""},
{""id"": 17, ""date"": ""2016-03-31""},
{""id"": 18, ""date"": ""2016-04-01""}]
```
Then to make one of them invalid:
sqlite-utils test-dates.db ""update dates set date = '//' where id = 10""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates,