Suggestions on mechanism or existing code - maintain persistence of file download history
On 31/01/20 9:53 PM, R.Wieser wrote:
>> Using ctrl+c is a VERY BAD idea.
> To have it just exit the program ? Yes, indeed.
> Though you /could/ keep track of what needs to be finished and have the
> ctrl-c handler do that for you (barf).
> Another posibility is to capture the ctrl-c and set a flag, which than
> instructs the program to terminate loops wherever possible - and thus have
> the program finish as if there was no more to do.
>> (see also 'sledgehammer to crack a nut')
> While I agree with you there, I've been searching for other ways to detect a
> keypress (in a console-based script) and have found none. IOW, you do not
> (seem to) have another option.
Color me disappointed! I was hoping to be enlightened as to how one
might code exactly that. (so I went looking...)
My first thought was a try...except looking for a keyboard error, but
wouldn't that only work if there was an input() loop? The idea of
putting, effectively the entire code-base, into a try...except block
seemed crude and likely to cause side-effects.
put me on a good path. So, I opened a door in my memory, shut my ears to
the intense screeching noise, and used a yard-broom to sweep away all
the rust that flaked off the hinges...
PSL's signal ? Set handlers for asynchronous events
(https://docs.python.org/3/library/signal.html) appears to handle
keyboard interrupts, *at least in MS-Windows*.
Rusty? I think, years ago, I used signal(s) to establish time-limits on
systems that had a possibility of 'run-away' or over-refining a solution
which would only ever be asymptotic. I've not used the library in this
That said, I'm using one of Python's asynchronous features, and Internet
activity is riddled with delay-periods which make such retrievals ripe
for async (code) solutions...
>> Why do you need to abandon the process mid-way?
> Take your pick. Mostly because of "I need to leave *now*". Or perhaps
> because of an "you have X seconds before the device shuts down because of
> the battery being low" situation.
I (personally) cannot make use of the above. However, even if my
portable's battery threatens complete disaster, the system "hibernates",
and thus averts major disaster, eg file system (or RDBMS) corruption.
This seems enough for 99% of 'average uses'. Of course it is regarded as
trivially-unacceptable in the non-stop world, eg banking. Would our OP
classify this project as anything other than the first of those
>> What is the OP's definition of "unlikely" or "acceptable risk"?
> Good question, but one I had no wish for to try to ascertain. I just gave
> some "worst case" secenario based replies (aka, the "did you think of <this>
> situation" kind).
Understood. However, your understanding of RDBMS interrupt 'security'
appears either aged, or biased to a particular implementation (which (in
my imagination) wouldn't match the high-standards I expect you hold).
Have already admitted that I (personally) may 'over use' RDBMS because
it is 'easy' for me. When we do feel the justification to examine the
details of "worst case scenarios"/"did you think of this?" RDBMS
stand-out as a 'solution' simply because of the incredible amounts of
time and depths of thought, highly-specialised minds (far superior to
any input I could offer) have invested in providing highly refined
services/servers. (which I dare-say informed @Chris' comments)
Neither MSFT nor the relevant GNU/Linux projects assure common file
systems to such a degree (and particularly not in the mentioned
circumstances). That's why there are FS 'rescue' procedures (Linux runs
fsck automatically when 'trouble' is noted during system-start and every
so-many reboots, regardless).
I'm sufficiently intrigued to wonder if anyone will dispute/correct such
- but not sufficiently motivated to research any position other than the
one currently held - so call me "Curious", but not "George" then...
>> There are so many reasons why such won't work first-time, when they should
>> every time; that it may be quite difficult to detect 'corruption'
> :-) That was not even considered. Just the "what do I still need to
> download" datafile.
Yes, we left the OP's (actual) spec, way-back somewhere!
>> Accordingly, there is no non-atomic transaction in the proposal
> The problem there is that you are second-guessing to what the OP will be
> writing, and that it will be good. I didn't and I don't assume that. I've
> seen too much "it works, so its good" noobie code. :-)
Absolutely! One of the 'joys' of working text-only is that even when one
knows (much) more about the background to a posted-question, it is still
so easy to 'answer-wrong'.
Even in my courses, where the Discussion Lists (are supposedly) related
to each week/chapter of a course, you'd think we'd have adequate 'hints'
as to the level of the trainee and where his/her question is
'coming-from'. Not true!
The secret is probably somewhere in the understanding of *that*
potential problem and going from there...
Sigh, if I had a dollar for every time someone looked at a simple
solution (MS-Excel I'm looking at you!) and uttered 'suggestions' such
as "we could run the entire company off one of these spreadsheet things"
or "if we captured all the company's transactions, couldn't we run it
off one Python pgm?"...
The world has moved-on though. Forty years ago it was common for someone
writing his/her first program(me) to come 'crying' "the computer made a
mistake" because it performed some simple arithmetic/statistical
calculation without yielding the expected answer!
That said, it continues to astonish me that computing-professionals
still can't apply the 'attention to detail' coding requires to their
typing - regardless of whether the input is code or a Discussion List
post. What is that mind-set???
(this comment NOT aimed at you, or any other participant on this list,
>> Do I want to deal with the complexities of managing files and corruptions,
>> in that arena?
>> Do you?
> :-) Its was-and-is not my choice to make. I gave the OP some stuff to
> think about, and left making the choice upto him. All three of the
> presented options are viable.
Good stuff! Yet at the same time, that very lack of clarity is good
reason to allow that 'another' solution might be 'right' - even if it is
(for any definitions of "right" or "better", or ...)
As sent to the OP. I appreciate these discussions, in the expectation of
learning something-new. (and with rust-removal paints at the ready!)
>> Be aware that formation rules for URLs are not congruent with OS FS rules!
> Yup. Though I seldom see that happen. Though I guess I should have
> mentioned that ... :-|
I'd show you the marks where that came back to bite me once - but
fortunately for you, the list doesn't allow graphic attachments (nor
should distressed (and distressing) images of such parts of one's
anatomy EVER be seen, even by qualified medical professionals...)