{"html_url": "https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1258437060", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490", "id": 1258437060, "node_id": "IC_kwDOCGYnMM5LAjnE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-26T18:24:44Z", "updated_at": "2022-09-26T18:24:44Z", "author_association": "OWNER", "body": "Just saw your great write-up on this: https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1382457780, "label": "Ability to insert multi-line files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1257072258", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490", "id": 1257072258, "node_id": "IC_kwDOCGYnMM5K7WaC", "user": {"value": 6180701, "label": "jeqo"}, "created_at": "2022-09-24T22:01:05Z", "updated_at": "2022-09-24T22:01:05Z", "author_association": "NONE", "body": "For completeness, the regex requires a bit more dark magic to capture the following lines, here is a _working_ expression: https://regex101.com/r/rsuEcs/1\r\n\r\n```\r\nsqlite-utils insert /tmp/log.db log multiline.log --text --convert \"\r\nimport re\r\n\r\nr = re.compile(r'^(?P<datetime>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2})(?:\\:\\s)(?P<log>(.*\\s\\s.*|.*)+)', re.MULTILINE)\r\n\r\ndef convert(text):\r\n    return [m.groupdict() for m in r.finditer(text)]\r\n\"\r\n```\r\n\r\n```\r\nBEGIN TRANSACTION;\r\nCREATE TABLE [log] (\r\n   [datetime] TEXT,\r\n   [log] TEXT\r\n);\r\nINSERT INTO \"log\" VALUES('2022-03-01T12:04:52','Here is a log message\r\n  that spans multiple lines');\r\nINSERT INTO \"log\" VALUES('2022-03-01T12:04:52','This is a single line');\r\nINSERT INTO \"log\" VALUES('2022-03-01T12:04:52','Here is another message\r\n  that spans multiple lines');\r\nCOMMIT;\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1382457780, "label": "Ability to insert multi-line files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1257063174", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490", "id": 1257063174, "node_id": "IC_kwDOCGYnMM5K7UMG", "user": {"value": 6180701, "label": "jeqo"}, "created_at": "2022-09-24T20:50:15Z", "updated_at": "2022-09-24T20:50:15Z", "author_association": "NONE", "body": "\ud83e\udd2f this is beautiful. Thanks @simonw !", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1382457780, "label": "Ability to insert multi-line files"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1256428818", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490", "id": 1256428818, "node_id": "IC_kwDOCGYnMM5K45US", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-23T16:37:58Z", "updated_at": "2022-09-23T16:38:35Z", "author_association": "OWNER", "body": "It should be possible to achieve this with the `--text` option: https://sqlite-utils.datasette.io/en/stable/cli.html?highlight=text#convert-with-text\r\n\r\nGiven an example like this in `multiline.log`:\r\n```\r\n2022-03-01T12:04:52: Here is a log message\r\n  that spans multiple lines\r\n2022-03-01T12:04:52: This is a single line\r\n2022-03-01T12:04:52: Here is another message\r\n  that spans multiple lines\r\n```\r\nYou should be able to run something like this:\r\n```\r\nsqlite-utils insert /tmp/log.db log multiline.log --text --convert \"\r\nimport re\r\n\r\nr = re.compile(r'^(?P<datetime>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}):(?P<log>.*)', re.MULTILINE)\r\n\r\ndef convert(text):\r\n    return [m.groupdict() for m in r.finditer(text)]\r\n\"\r\n```\r\nAfter running this I get:\r\n```\r\nsqlite-utils rows /tmp/log.db log\r\n[{\"datetime\": \"2022-03-01T12:04:52\", \"log\": \" Here is a log message\"},\r\n {\"datetime\": \"2022-03-01T12:04:52\", \"log\": \" This is a single line\"},\r\n {\"datetime\": \"2022-03-01T12:04:52\", \"log\": \" Here is another message\"}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1382457780, "label": "Ability to insert multi-line files"}, "performed_via_github_app": null}