issues

Table actions

10 rows where repo = 197882382, state = "closed" and user = 9599 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: comments, updated_at, closed_at, created_at (date), updated_at (date)

type 2

state 1

  • closed · 10 ✖

repo 1

  • healthkit-to-sqlite · 10 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
975158266 MDU6SXNzdWU5NzUxNTgyNjY= 19 table activity_summary has no column named appleMoveTime simonw 9599 closed 0     0 2021-08-20T00:46:44Z 2021-08-20T00:54:34Z 2021-08-20T00:54:34Z MEMBER  

Got this error today against a fresh export:

table activity_summary has no column named appleMoveTime
healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/19/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
519038979 MDU6SXNzdWU1MTkwMzg5Nzk= 10 Failed to import workout points simonw 9599 closed 0     4 2019-11-07T04:50:22Z 2019-11-08T01:18:37Z 2019-11-08T01:18:37Z MEMBER  

I just ran the script and it failed to import any workout_points, though it did import workouts.

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
472097220 MDU6SXNzdWU0NzIwOTcyMjA= 7 Script uses a lot of RAM simonw 9599 closed 0     3 2019-07-24T06:11:11Z 2019-07-24T06:35:52Z 2019-07-24T06:35:52Z MEMBER  

I'm using an XML pull parser which should avoid the need to slurp the whole XML file into memory, but it's not working - the script still uses over 1GB of RAM when it runs according to Activity Monitor.

I think this is because I'm still causing the full root element to be incrementally loaded into memory just in case I try and access it later.

http://effbot.org/elementtree/iterparse.htm says I should use elem.clear() as I go. It also says:

The above pattern has one drawback; it does not clear the root element, so you will end up with a single element with lots of empty child elements. If your files are huge, rather than just large, this might be a problem. To work around this, you need to get your hands on the root element.

So I will try that recipe and see if it helps.

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
472104705 MDExOlB1bGxSZXF1ZXN0MzAwNTgwMjIx 8 Use less RAM simonw 9599 closed 0     0 2019-07-24T06:35:01Z 2019-07-24T06:35:52Z 2019-07-24T06:35:52Z MEMBER dogsheep/healthkit-to-sqlite/pulls/8

Closes #7

 healthkit-to-sqlite 197882382 pull     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/8/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 0  
470691622 MDU6SXNzdWU0NzA2OTE2MjI= 5 Add progress bar simonw 9599 closed 0     2 2019-07-20T16:29:07Z 2019-07-22T03:30:13Z 2019-07-22T02:49:22Z MEMBER  

Showing a progress bar would be nice, using Click.

The easiest way to do this would probably be be to hook it up to the length of the compressed content, and update it as this code pushes more XML bytes through the parser:

https://github.com/dogsheep/healthkit-to-sqlite/blob/d64299765064501f4efdd9a0b21dbdba9ec4287f/healthkit_to_sqlite/utils.py#L6-L10

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
470856782 MDU6SXNzdWU0NzA4NTY3ODI= 6 Break up records into different tables for each type simonw 9599 closed 0     1 2019-07-22T01:54:59Z 2019-07-22T03:28:55Z 2019-07-22T03:28:50Z MEMBER  

I don't think there's much benefit to having all of the different record types stored in the same enormous table. Here's what I get when I use _facet=type:

I'm going to try splitting these up into separate tables - so HKQuantityTypeIdentifierBodyMassIndex becomes a table called rBodyMassIndex - and see if that's nicer to work with.

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
470637152 MDU6SXNzdWU0NzA2MzcxNTI= 2 Import workouts simonw 9599 closed 0     1 2019-07-20T05:20:21Z 2019-07-20T06:21:41Z 2019-07-20T06:21:41Z MEMBER  

From #1

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
470640505 MDU6SXNzdWU0NzA2NDA1MDU= 4 Import Records simonw 9599 closed 0     1 2019-07-20T06:11:20Z 2019-07-20T06:21:41Z 2019-07-20T06:21:41Z MEMBER  

From #1: python 'Record': {'attr_counts': {'creationDate': 2672233, 'device': 2665111, 'endDate': 2672233, 'sourceName': 2672233, 'sourceVersion': 2671779, 'startDate': 2672233, 'type': 2672233, 'unit': 2650012, 'value': 2672232}, 'child_counts': {'HeartRateVariabilityMetadataList': 2318, 'MetadataEntry': 287974}, 'count': 2672233, 'parent_counts': {'Correlation': 2, 'HealthData': 2672231}},

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
470637206 MDU6SXNzdWU0NzA2MzcyMDY= 3 Import ActivitySummary simonw 9599 closed 0     0 2019-07-20T05:21:00Z 2019-07-20T05:58:07Z 2019-07-20T05:58:07Z MEMBER  

From #1

python 'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980, 'activeEnergyBurnedGoal': 980, 'activeEnergyBurnedUnit': 980, 'appleExerciseTime': 980, 'appleExerciseTimeGoal': 980, 'appleStandHours': 980, 'appleStandHoursGoal': 980, 'dateComponents': 980}, 'child_counts': {}, 'count': 980, 'parent_counts': {'HealthData': 980}},

 healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/3/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
470637068 MDU6SXNzdWU0NzA2MzcwNjg= 1 Use XML Analyser to figure out the structure of the export XML simonw 9599 closed 0     1 2019-07-20T05:19:02Z 2019-07-20T05:20:09Z 2019-07-20T05:20:09Z MEMBER   healthkit-to-sqlite 197882382 issue     
{
    "url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);